https://llmobile.news/categories/hardware/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.news/ticker/llama-cpp-hexagon-npu-snapdragon/https://llmobile.news/ticker/llama-cpp-hexagon-npu-snapdragon/Thu, 10 Sep 2026 11:00:00 +0200A user has reported results from llama.cpp’s Hexagon NPU backend on a OnePlus 12 with a Snapdragon 8 Gen 3. They cross-compiled llama.cpp on Ubuntu following the project’s Snapdragon documentation and ran it from Termux on the phone. The reported figures are 8 tokens per second of prompt processing and 4.5 of generation with gemma-3-12b-it-qat at Q4_0, and 20 tokens per second of prompt processing and 12.5 of generation with gemma-3-4b-it-qat at Q4_0. According to the report the speed is about the same as running on the same chip’s CPU, but the phone does not get hot. The report lists several limitations. The backend accepts only Q4_0, IQ4_NL, MXFP4, Q8_0 and F32 files, and does not support KV cache quantisation. It runs on chips as old as the Snapdragon 888, but the poster notes that only the 8 Gen 2 and newer carry the tensor module used for language models. The Hexagon NPU can address 4 GB, so a model plus KV cache beyond that has to be split across more than one NPU device: LD_LIBRARY_PATH=./lib:/vendor/lib64 \ ADSP_LIBRARY_PATH=./lib \ GGML_HEXAGON_NDEV=2 \ ./bin/llama-completion \ -m /sdcard/gguf/gemma-3-12b-it-qat-Q4_0.gguf \ -ngl 99 --device HTP0,HTP1 \ -sys 'You are a helpful AI assistant' For context the poster cites 34 INT8 TOPS and 76.8 GB/s of memory bandwidth for the 8 Gen 3 NPU, against 80 TOPS and 228 GB/s for Qualcomm’s X2 Elite Extreme and 248 TOPS and 936 GB/s for an Nvidia RTX 3090. They ask readers with a Snapdragon 8 Elite Gen 5 phone or an X2 Elite Extreme laptop to post their own numbers, since those parts are said to address more than 4 GB and would not need multiple devices. Source: https://www.reddit.com/r/LocalLLaMA/comments/1t0k6fj/running_llamacpp_on_snapdragon_hexagon_npu_seems/ Read the article: https://llmobile.news/ticker/llama-cpp-hexagon-npu-snapdragon/QualcommAndroidNPUllama.cppGemmahttps://llmobile.news/ticker/iphone-18-pro-a20-pro-neural-engine/https://llmobile.news/ticker/iphone-18-pro-a20-pro-neural-engine/Wed, 09 Sep 2026 19:00:00 +0200Apple has introduced the iPhone 18 Pro and iPhone 18 Pro Max with the A20 Pro chip. According to Apple, the chip has a new dual 16-core Neural Engine, 32 cores in total, which the company describes as double the AI processing power of A19 Pro. Apple says the Neural Engine accelerates on-device AI models and computational photography. Image: Apple. The A20 Pro also offers 50 percent more memory bandwidth than A19 Pro, and its 6-core CPU includes integrated Neural Accelerators. Apple positions the chip for “more advanced on-device AI workloads” alongside games. The phones ship with iOS 27, Apple Intelligence and the new Siri AI. Apple states that Apple Intelligence uses on-device processing together with Private Cloud Compute. Pre-orders start on 12 September, with availability from 18 September; prices start at $1,199 for the iPhone 18 Pro and $1,299 for the iPhone 18 Pro Max, both with 256 GB. Source: https://www.apple.com/newsroom/2026/09/apple-debuts-iphone-18-pro-and-iphone-18-pro-max/ Read the article: https://llmobile.news/ticker/iphone-18-pro-a20-pro-neural-engine/AppleiPhoneChipsNPUhttps://llmobile.news/ticker/pixel-watch-5-offline-gemini/https://llmobile.news/ticker/pixel-watch-5-offline-gemini/Wed, 12 Aug 2026 19:30:00 +0200Gemini Intelligence is coming to the Pixel Watch 5, 9to5Google reports. According to the report, offline Gemini commands on the watch use a separate on-device model when the phone or an internet connection is unavailable. Those commands cover timers and alarms, brightness and modes, music control, opening apps and starting workouts. On-device smart replies now offer three responses instead of one, which the report attributes to a Gemini Nano upgrade that makes them 50 percent faster. Other parts of the feature set depend on a connection. Proactive Suggestions, formerly Magic Cue, are generated on a paired Pixel 11 and bridged to the watch, and Personal Intelligence draws on Gmail, Calendar and Keep. The update also brings a new At a Glance space on the watch face for timers, workouts, music, flight details and navigation. Source: https://9to5google.com/2026/08/12/pixel-watch-5-gemini-intelligence/ Read the article: https://llmobile.news/ticker/pixel-watch-5-offline-gemini/GooglePixelWearablesGemini Nanohttps://llmobile.news/ticker/pixel-11-tensor-g6/https://llmobile.news/ticker/pixel-11-tensor-g6/Wed, 12 Aug 2026 19:00:00 +0200Google has announced the Pixel 11, Pixel 11 Pro and Pixel 11 Pro XL, built around the Tensor G6 chip. Google states that Tensor G6 packs 50 percent more TPU compute and, paired with the latest Gemini Nano model, processes on-device AI tasks up to 3.5 times faster while using up to 3.5 times less energy. The company also cites an upgraded CPU with 25 percent faster web browsing and 15 percent quicker app launches, and says the chip powers the 30x Super Zoom on the 5x telephoto lens. Google does not publish RAM figures, model sizes or per-task latency in the announcement. Pre-orders opened on 12 August, with retail availability from 20 August. ▶Meet Google Pixel 11 ProLoading connects your browser to www.youtube-nocookie.com, which may process your IP address and use cookies. Load contentOpen externallyVideo: Made by Google. Source: https://blog.google/products-and-platforms/devices/pixel/google-pixel-11-pro-xl/ Read the article: https://llmobile.news/ticker/pixel-11-tensor-g6/GooglePixelChipsNPUGemini Nanohttps://llmobile.news/ticker/qualcomm-amon-agents-new-app/https://llmobile.news/ticker/qualcomm-amon-agents-new-app/Tue, 16 Jun 2026 07:08:00 +0200Qualcomm chief executive Cristiano Amon says the company is working on more than 40 designs of new AI devices, CNBC reports from an interview on its “The Tech Download” podcast. He listed jewellery, earbuds with cameras, pins and watches among the form factors. Amon described a shift in what devices are organised around: The phone is around the agent. The new classes of devices … are going to be around the agent as well. And the agent will be the one that will understand human intentions and will do things for you, so there is a shift in what the center of gravity is. He said phones will not disappear. On software, he told CNBC that apps are “not dead” but “are going to change”, adding: “Those agents are going to be the new app.” His example was an agent that retrieves banking transaction details without the user navigating the app. On smart glasses, Amon put current shipments in the “order of multiple tens of millions” a year and said that in “a couple of years” the category could reach the “order of hundreds of millions of glasses and could become as big as smartphones”. CNBC cites Counterpoint Research figures of 1.26 billion smartphones shipped in 2025, about 3 percent above the prior year. Amon linked the entry of AI companies into hardware, such as OpenAI’s purchase of Jony Ive’s startup io, to both agent endpoints and training data, saying the devices will gather data on a scale that is “exponentially larger” than the data used to train current models. He said Qualcomm’s chip roadmap is being reworked in response: “An entire roadmap, because I believe none of the devices we have today are prepared for the future.” The full interview is available as a video on CNBC. Source: https://www.cnbc.com/2026/06/16/qualcomm-ceo-ai-devices-agents.html Read the article: https://llmobile.news/ticker/qualcomm-amon-agents-new-app/QualcommAgentsWearablesSmart glasseshttps://llmobile.news/ticker/llm-inference-at-the-edge/https://llmobile.news/ticker/llm-inference-at-the-edge/Tue, 24 Mar 2026 19:28:00 +0100Four authors at Conscious Engines published a benchmark of Qwen 2.5 1.5B at 4-bit across four edge platforms on March 24, 2026, feeding each device the same 258-token prompt 20 times in a row with a 1-second gap instead of once. On an iPhone 16 Pro the first two runs peak at 40.49 tokens per second, and from run 17 onwards the phone holds a plateau of 23.67 tok/s, which the authors put at 41.5 percent below that peak. The iPhone passes through the three thermal states iOS reports to an app. Runs 1 and 2 sit in Normal and average 36.83 tok/s, runs 3 to 16 sit in Warm and average 24.96 tok/s, and runs 17 to 20 sit in Hot at 23.67 tok/s with a variation of 1.2 percent. The authors write that the 1-second gap allows no thermal recovery, and they measure 5 percent of battery drained over the 20 runs, which they project to roughly 400 replies of this length per charge. A Samsung Galaxy S24 Ultra degrades more gently in the same test. Its Adreno 750 GPU boosts to 1,000 MHz on runs 4 and 5 for a peak of 12.21 tok/s, then steps down to a 720 to 770 MHz plateau from run 8 where the authors measure 10.38 tok/s, 15 percent off the peak, with a low of 9.55 tok/s on the final run. They stress that reaching this plateau took deliberate tuning, with the display off, the context capped at 2,048 tokens and MLC-LLM’s prefill chunk cut to 128 tokens, and that earlier unconstrained runs hit an OS-enforced GPU frequency floor. Neither phone ran the model on its NPU. The paper states that MLX dispatches to the A18 Pro GPU through Metal and does not target Apple’s Neural Engine, and that the S24 Ultra ran through MLC-LLM on the Adreno GPU, leaving the Snapdragon 8 Gen 3’s Hexagon NPU outside the measurement. The one NPU that did run the model, a Hailo-10H module in a Raspberry Pi 5, was also the only part that never throttled, holding 6.914 tok/s across all 20 runs with a variation of 0.04 percent at under 2 W, its CPU at 52.7°C and the NPU at 58.5°C, with attention layers that hailo-ollama does not offload falling back to the Pi’s Cortex-A76 CPU. A user running llama.cpp’s Hexagon NPU backend on a Snapdragon 8 Gen 3 phone reported 12.5 tok/s of generation on Gemma 3 4B and said the phone stayed cool. A laptop RTX 4050 on battery led on speed at 131.7 tok/s and 34.12 W, its GPU climbing from 55°C to 70°C without throttling. Energy per token comes out at 270.5 mJ for the Hailo module measured across the whole system, 297.3 mJ for the RTX 4050 measured at the GPU, and 143.0 mJ for the S24 Ultra plateau derived from the Android fuel gauge with the display off. The authors state that the three readings differ in scope and call the resemblance suggestive rather than settled, and report no energy figure for the iPhone because iOS exposes no per-component power reading. They tested one model, one prompt and one unit per platform across four different runtimes, and describe the results as characterising hardware and software together rather than hardware on its own. Source: https://arxiv.org/abs/2603.23640 Read the article: https://llmobile.news/ticker/llm-inference-at-the-edge/BenchmarksNPUiPhoneSamsungResearchhttps://llmobile.news/ticker/intelligence-per-watt/https://llmobile.news/ticker/intelligence-per-watt/Tue, 11 Nov 2025 07:33:00 +0100Researchers at Stanford University and Together AI profiled more than 20 local models on 1M real queries and report that at least one model with 20B or fewer active parameters answered 88.7% of them correctly. The best single model in their set, GPT-OSS-120B, reached 71.4% on its own, averaged across the four query collections. Their 2025 efficiency figure comes from that model running on an Apple M4 Max. The paper proposes intelligence per watt, mean task accuracy divided by the mean power the accelerator draws while answering, so a pairing that gets more answers right on the same power budget scores higher. The best local pairing each year moved from 7.92×10⁻⁴ accuracy per watt for Mixtral 8x7B on an NVIDIA Quadro RTX 6000 in 2023, to 1.80×10⁻³ for Llama 3.1 8B on an RTX 6000 Ada in 2024, to 4.18×10⁻³ for GPT-OSS-120B on an Apple M4 Max in 2025, which the authors put at a 5.3 times gain over two years. Query coverage for that single best pairing rose from 23.2% to 71.3% across the same three years. A companion metric divides accuracy by the energy a query consumes instead, so slow generation counts against a device. Diagram: Saad-Falcon et al., Figure 1 of the paper, licensed CC BY 4.0. The hardware panel separates the accelerators the authors treat as local from the cloud parts. The 1M queries come from four collections. The team took 500K prompts from WildChat, a month of real ChatGPT traffic, 500K from NaturalReasoning, 12K from MMLU Pro and 26.5K from SuperGPQA, all single-turn. WildChat has no ground truth, so GPT-4o judged whether a local model’s answer matched one written by Qwen3-235B, while the other three collections supplied reference answers. Every run used a batch size of 1, one query at a time. The hardware the authors call local is desktop and workstation class rather than mobile. Their 2025 numbers ran on an Apple Mac Studio with an M4 Max, 128 GB of unified memory and 480 W of continuous power, and the 2023 and 2024 points used NVIDIA workstation cards. No phone, tablet or wearable was profiled. Running the same Qwen3 models, the M4 Max reached 1.40 times lower accuracy per watt than an NVIDIA B200 and 1.6 to 2.3 times lower accuracy per joule, a gap the authors read as headroom for local accelerator design. For the savings case the team simulated 80.2M queries over 24 hours split between four small models on M4 Max machines and Qwen3-235B in the cloud. Perfect routing to the smallest capable model cut energy by 80.4% against sending everything to the frontier model, and a router that picks correctly 80% of the time cut it by 64.3%, with the perfect case described as a theoretical maximum. The authors note that coverage falls in technical domains such as architecture and engineering, that GPT-OSS-120B answered 88.9% of WildChat queries against 64.9% of NaturalReasoning ones, and that they release their profiling harness so the measurements can be repeated on new hardware. Source: https://arxiv.org/abs/2511.07885 Read the article: https://llmobile.news/ticker/intelligence-per-watt/ResearchBenchmarksApple SiliconNvidiaQwenhttps://llmobile.news/ticker/a19-pro-neural-accelerators/https://llmobile.news/ticker/a19-pro-neural-accelerators/Tue, 09 Sep 2025 20:00:00 +0200Apple announced the A19 Pro alongside the iPhone 17 Pro on September 9, 2025. Its 6-core GPU carries Neural Accelerators built into each core, which Apple says work together with the 16-core Neural Engine to power AI models, graphics and games. The company states that the chip enables running large local language models on the phone. The A19 Pro also has a 6-core CPU that Apple calls the fastest in any smartphone, along with a larger cache and more memory than the A18 Pro. Combined with a vapor chamber cooling system, Apple puts sustained performance at up to 40 percent above the previous generation. Image: Apple. The vapor chamber Apple credits for the sustained performance gain. Source: https://www.apple.com/newsroom/2025/09/apple-unveils-iphone-17-pro-and-iphone-17-pro-max/ Read the article: https://llmobile.news/ticker/a19-pro-neural-accelerators/AppleiPhoneChipsApple SiliconNPUhttps://llmobile.news/ticker/shadownpu/https://llmobile.news/ticker/shadownpu/Fri, 22 Aug 2025 09:41:00 +0200Researchers from Peking University and BUPT report up to 4.5 times faster end-to-end inference on a Xiaomi 14 with a Snapdragon 8 Gen 3, and 3.5 times faster on average for the attention kernel alone, with a peak of 6.9 times. The baseline is the arrangement most on-device frameworks use today, float32 full attention running on the CPU or GPU while the rest of the model sits on the NPU. The authors measured Qwen2-0.5B, Qwen2-1.5B, PhoneLM-0.5B and PhoneLM-1.5B on summarisation and two mobile agent tasks, and gave their own system only one of the phone’s middle CPU cores. Against the native attention of llm.npu, which uses four CPU cores or the GPU, they report up to 3.0 times lower latency from that one core. The paper is due at MobiSys 2026. The starting problem is that attention keeps falling back off the NPU. Attention multiplies three activation tensors together, and mobile NPUs compile a static graph that fixes one quantisation scale factor for a whole tensor, so the authors measure an average 18 percentage point accuracy drop when the operator runs in INT8 on the NPU. Their trade is to stop asking the NPU for the answer and ask it only for the ranking. The NPU computes the query-key scores in INT8 purely to find which tokens matter, which the authors report recovers over 99 percent of the positions a float run would pick, since picking the largest values needs relative magnitudes rather than exact ones. The CPU then computes real attention in float over roughly 20 percent of the tokens and throws the other 80 percent away, at an average accuracy cost of 0.4 percentage points against the float baseline. Each attention head gets its own sparsity ratio, fixed by an offline profiling run the paper puts at about five minutes on a single A100. On a Redmi K60 Champion Edition with a Snapdragon 8 Gen 2 the authors read battery voltage and current from the kernel and report a single attention kernel at 1024 tokens dropping from 8.59 J to 1.12 J for PhoneLM-1.5B, a factor of 7.66. Speedups on that older phone are smaller, 2 times, 1.25 times and 1.22 times across the three datasets, which the authors attribute to its Hexagon V73 NPU against the V75 in the Xiaomi 14. The paper reports no memory footprint figures. It does report that end-to-end inference time on the Xiaomi 14 stayed between 14.5 and 15.5 seconds while the phone was also taking photos, playing video, playing music or running a shopping app, since the LLM work is not competing for the cores those apps use. Diagram: Yin et al. The authors state a prototype of more than 10,000 lines of C++ and Python, built on Qualcomm’s QNN and Hexagon SDK for the NPU side and integrated into llm.npu, which they call the only fully open-source framework running on NPUs. The paper names no repository and announces no artefact release, and the title calls the system ShadowNPU while the text throughout calls the module shadowAttn. The arXiv listing carries a CC BY 4.0 license. For a hands-on point of comparison on the same silicon family, a user test of llama.cpp’s Hexagon NPU backend on a Snapdragon 8 Gen 3 phone put generation at about the same speed as the chip’s CPU, without the heat. Source: https://arxiv.org/abs/2508.16703 Read the article: https://llmobile.news/ticker/shadownpu/QualcommNPUResearchQwenAndroidhttps://llmobile.news/ticker/roma-qlora-accelerator/https://llmobile.news/ticker/roma-qlora-accelerator/Mon, 17 Mar 2025 10:44:00 +0100Researchers at Shanghai Jiao Tong University, Peking University and Microsoft Research Asia described ROMA on March 17, 2025, an accelerator design that holds a quantised language model in on-chip read-only memory. The paper reports results from Verilog synthesis against TSMC’s 7 nm standard cell library, not from a fabricated chip and not from an FPGA prototype, so every speed, area and power figure in it is a toolchain estimate rather than a measurement on hardware. On that basis the authors report peak decoding of 31,800 tok/s for a 4-bit 3B Llama model with an empty KV cache. The design splits storage by how often the weights change. The frozen base model, quantised to 2-bit or 4-bit, sits in ROM, which cannot be rewritten once the chip has been manufactured, while the small trainable LoRA adapter and the KV cache sit in SRAM, which can. The authors argue that a base model is stable enough for that, with updates expected over years rather than months, and that adaptation to new tasks happens in the adapter instead. What ROM buys is density, and the paper puts a ROM bit cell at roughly a third of the area of an SRAM bit cell in TSMC’s 7 nm memory compiler, enough to fit the whole quantised model on chip without external memory. The synthesised configuration carries 1.86 GB of ROM and 304 MB of SRAM at 500 MHz, with an area of 503.7 mm² and power of 33.1 W, the area and latency coming from Synopsys Design Compiler and the power from Synopsys PrimeTime PX. It holds either a 4-bit 3B or a 2-bit 8B Llama model entirely on chip, and for the 3B model with a rank-16 adapter the authors report a time to first token of 5.6 ms at 256 input tokens and 140.2 ms at 4K, with decoding staying above 10,000 tok/s once 4K tokens are cached. They also compare against measured hardware, an Intel i5-1135G7 running llama.cpp at 6.8 tok/s and an Nvidia RTX 4090 running TensorRT-LLM at 219 tok/s on the same 4-bit 3B model, against 20,078 tok/s for the ROMA estimate at a 256-token input. Most of the paper’s engineering goes into shrinking the ROM. A structure the authors call B-ROM stores the constant bits in the chip’s wiring instead of in transistors, which cuts the transistor count to about a quarter and the ROM area by around 40 percent against a standard ROM, and a fused cell layout places that wiring-heavy structure in the same physical area as the transistor-heavy compute logic. The paper states no tapeout, no fabrication schedule and no cost figures, and it names the fixed base model as the price of the approach, since ROM content is set at manufacture. It is posted under the arXiv perpetual non-exclusive license. Source: https://arxiv.org/abs/2503.12988 Read the article: https://llmobile.news/ticker/roma-qlora-accelerator/ChipsMemoryQuantisationResearchLlamahttps://llmobile.news/ticker/on-device-inference-tradeoffs/https://llmobile.news/ticker/on-device-inference-tradeoffs/Wed, 12 Mar 2025 08:01:00 +0100Maximilian Abstreiter, Sasu Tarkoma and Roberto Morabito of the University of Helsinki and EURECOM measured 11 language models from 0.5B to 9.2B parameters on a Raspberry Pi 5 and an NVIDIA Jetson Orin Nano Developer Kit, both with 8 GB of RAM, running every model through llama.cpp at 4-bit. On the Pi, the authors report that only models up to Llama 3.2 3B, or up to Phi 3.5 mini in the faster of the two 4-bit formats they tested, generated text above 5.3 tokens per second, the rate they derive from average English reading speed. The Orin’s GPU stayed above that rate for all 11 models, its CPU cores up to InternLM 2.5 7B in that same format. Generation speed drops off steeply with model size in the paper’s charts. Qwen 2 0.5B runs at roughly 46 tokens per second on the Orin’s CPU cores and roughly 33 on the Pi, while Phi 3.5 mini at 3.8B parameters sits at about 5 on the Pi, about 10 on the Orin CPU and about 15 on the Orin GPU. The authors also contradict Meta’s claim that Llama 3.2 is optimised for Arm processors, since the advantage they saw for Llama 3.2 1B showed up in GPU inference and not on either CPU. Memory is where the measurements turn hardest. Peak usage grows close to linearly with model size, reaching about 7.9 GB for Gemma 2 9B on the Pi, and it flattens out on the Orin above Phi 3.5 mini because under 7 GB of the nominal 8 GB is usable once the operating system and background processes take their more than 1 GB. Raising the context window to 1028 tokens pushed the largest model into constant page faults on the CPU and made the Orin crash and restart during GPU inference. First-run load times ran from 5 seconds to over 200 seconds, and for Yi 1.5 6B and up the faster 4-bit format raised load times by 17 to 53 times, because those models get evicted from memory and re-read from the microSD card for every inference. Scaled to the 18.84 Wh battery of a Samsung Galaxy S24 Ultra at full utilisation, the March version of the paper puts Gemma 2 9B at 193 inference runs on the Pi and 101 on the Orin GPU, against 3,928 and 4,280 runs for Qwen 2 0.5B, with no configuration lasting 7 hours. The setup was built to keep thermal throttling out of the results rather than to observe it, so the authors fitted the Pi with an add-on heatsink after it overheated in early experiments, ran the fans on both boards at full speed and left 30 seconds between runs. Quantisation cost measurable quality. Perplexity, which tracks how uncertain a model is about its next token, rose against the 16-bit weights for every model and both 4-bit formats and rose most sharply for the smallest models, while the downstream benchmarks showed almost no accuracy drop. The paper is on arXiv and under review at an ACM journal. Update, November 20, 2025. The authors revised the paper after validating their power measurements against external hardware, which raised the measured consumption. The battery estimate for Gemma 2 9B falls from 193 runs to 75 on the Pi and from 101 to 73 on the Orin GPU, and no configuration now reaches 3.5 hours at full utilisation. The speed, memory and quality results are unchanged. Source: https://arxiv.org/abs/2503.09114 Read the article: https://llmobile.news/ticker/on-device-inference-tradeoffs/ResearchBenchmarksllama.cppQuantisationNvidiahttps://llmobile.news/ticker/genai-at-the-edge-survey/https://llmobile.news/ticker/genai-at-the-edge-survey/Wed, 19 Feb 2025 21:43:00 +0100Seven researchers at Johns Hopkins University and Duke University published a survey of generative AI on edge devices on February 19, 2025. Most of the accelerator hardware it collects has not been built. Its table of 12 accelerators for language and diffusion models covers 8 simulator studies against 4 fabricated chips, and the highest peak energy efficiency in it is 74.34 TOPS/W, from a 28nm computing-in-memory chip designed for Stable Diffusion. Compression is the part the survey treats as settled enough to quantify, and it attributes every figure to the work it reviews. It reports unstructured pruning reaching up to 60 percent sparsity on large models and a sparsity-aware inference system turning that into a 3 times speedup, while structured pruning that removes whole components gives about 2 times with little quality loss. Quantisation has been taken to 1 bit per weight in work such as BitNet, and the survey names what is still open there as holding generation quality under extreme compression and finding efficient ways to train quantised models on the edge devices themselves rather than only to run them. The frameworks chapter is where the survey names hardware a reader can buy. It cites a measurement that applying NVIDIA’s TensorRT to transformer models on a Jetson Xavier cuts latency by over 60 percent with negligible accuracy loss, and points to MELTing Point, a benchmark suite that tracks language model energy use and memory footprint across smartphones and Jetson boards. The authors state that no dedicated survey of generative AI at the edge existed before theirs, close by naming model personalisation and security across distributed edge nodes as the challenges that persist, and publish on arXiv under a Creative Commons Attribution 4.0 license. Diagram: Navardi and colleagues, from the survey. Source: https://arxiv.org/abs/2502.15816 Read the article: https://llmobile.news/ticker/genai-at-the-edge-survey/ResearchChipsQuantisationNPUhttps://llmobile.news/ticker/exolabs-benchmarks/https://llmobile.news/ticker/exolabs-benchmarks/Wed, 25 Dec 2024 19:55:00 +0100EXO Labs opened a public benchmark site on December 25, 2024, as the first entry in a twelve-part series, publishing generation speed and time to first token measured on real consumer devices. EXO states these are measurements rather than estimates. In the data behind the site, an iPhone 15 Pro with 8 GB of RAM generates 14.0 tok/s on Llama 3.1 8B and takes 1.67 seconds to return its first token. A Galaxy S24 Ultra with 12 GB manages 5.6 tok/s on the same 8B model, with 4.16 seconds to the first token, according to EXO’s published results. On the smaller Llama 3.2 1B the iPhone reaches 56.1 tok/s and the Galaxy 22.5 tok/s. A single Mac mini M4 Pro with 24 GB sits at 49.3 tok/s on Llama 3.1 8B, which puts the phone at under a third of the Mac’s rate on the same model. EXO splits a model into contiguous slices of its layers and hands each slice to a different device, so a model too large for one machine can run across several. The activations passed from one device to the next are small, under 4 KB for Llama 3.2 3B, and EXO says the bottleneck is network latency rather than bandwidth. Pooling phones does not speed up a single reply. A cluster of two iPhone 15 Pros and one Galaxy S24 Ultra runs Llama 3.1 8B at 7.0 tok/s, below the 14.0 tok/s of one iPhone alone, which EXO attributes to network overhead when the model already fits on a single device. Three Mac mini M4 Pros run Llama 3.3 70B at 4.1 tok/s, a model that does not fit in the 24 GB of any one of them. Every entry carries a timestamp, a git commit and the identifier of the automated run that produced it, and the site is free to browse. EXO says it is expanding the suite towards mixed device types and different quantisations, and the exo runtime itself is on GitHub under Apache 2.0. Source: https://blog.exolabs.net/day-1/ Read the article: https://llmobile.news/ticker/exolabs-benchmarks/BenchmarksiPhoneAndroidLlamaApple Siliconhttps://llmobile.news/ticker/small-model-training-bottlenecks/https://llmobile.news/ticker/small-model-training-bottlenecks/Fri, 25 Oct 2024 12:30:00 +0200Seven Apple researchers measured what it costs to train language models up to 2B parameters on rented Nvidia GPUs and found the most expensive card was the worst buy. The H100-80GB returned the fewest tokens per dollar of the three cards tested at every one of their four model sizes, 100M, 500M, 1B and 2B, while costing 2.53 times as much per hour as the A100-40GB. The price scale comes from normalising published hourly rates at Lambda Labs and Google Cloud, with the A100-40GB set at 1.00 and the A100-80GB landing at 1.46. The same comparison shows cost efficiency falling as GPUs are added, with a single card the best or near-best point at all four model sizes. For the 1B model, tokens per dollar drop from about 17,000 on one GPU to about 10,300 on 64. FlashAttention mattered more at small scale than at large. Reading the authors’ first figure, the 100M model gained roughly three times the tokens per dollar over vanilla attention, while at 2B the gain was under two times. The authors put this down to the cost of attention being quadratic in context length, so it dominates once the hidden dimension shrinks and the small models become bound by data movement between CPU and GPU and between GPUs. Vanilla attention also ran out of memory at a global batch size of 1024 on the 1B and 2B models, where FlashAttention did not. Two habits carried over from large-model training did not pay off. Plain Distributed Data Parallel, which keeps a full copy of the model on every GPU, beat both sharded FSDP variants for the smaller models because it moves less data, and FSDP only pulled ahead at 2B, where it kept training at a global batch size that made DDP run out of memory. Filling the cards was not cost-optimal either, since the batch size figure peaks around a global batch size of 128 for the 100M model, 32 for 500M, 16 for 1B and 8 for 2B, in every case well short of what the memory allowed. These are training runs on datacentre GPUs, not anything running on a phone, and the authors frame the work as guidance for labs with small budgets rather than for deployment. They name their own limits, running each setup at least three times for only 10 training steps, reporting normalised price ratios rather than absolute dollars, using no gradient accumulation, and leaving GPU utilisation and memory bandwidth measurements to future work. They also assume that settings such as the learning rate can be retuned to converge at whichever hardware configuration turns out cheapest. The 8-page paper is free to read on arXiv under the site’s non-exclusive distribution licence. Source: https://arxiv.org/abs/2410.19456 Read the article: https://llmobile.news/ticker/small-model-training-bottlenecks/AppleNvidiaResearchBenchmarksPyTorchhttps://llmobile.news/ticker/palmbench/https://llmobile.news/ticker/palmbench/Sat, 05 Oct 2024 05:37:00 +0200Researchers at the University of Wisconsin-Madison, Amazon Web Services AI and Uber ran quantised language models across eight phones and edge boards and measured the widest gap between Apple and everything else. In PalmBench, published as a conference paper at ICLR 2025, TinyLlama-1.1B at 4-bit reached close to 100 tok/s on an iPhone 15 Pro against roughly 30 tok/s on the fastest Android phone in the set, a Pixel 7. The authors report that the three-year-old iPhone 12 Pro also stayed ahead of every Android device and of Nvidia’s Jetson Orin Nano. Chart: Li et al., licensed CC BY 4.0. Throughput measured with MLC LLM, which was the only framework that ran on all eight platforms. The test bed covers a Pixel 4, Pixel 5a, Pixel 7, Galaxy S22 Ultra, iPhone 12 Pro, iPhone 15 Pro, an Orange Pi 5 and a Jetson Orin Nano, each with 6 GB or 8 GB of memory. Models include Llama-2-7B, Llama-3 and 3.1-8B, Llama-3.2-3B, Mistral-7B, Vicuna, Qwen2, Phi-2, Gemma-2-2B, RedPajama-INCITE-3B and TinyLlama-1.1B, quantised from 2-bit to 8-bit and additionally with GPTQ, AWQ and FasterTransformer. The team used two runtimes, MLC LLM and llama.cpp, and states that llama.cpp does not run on iPhone, so the cross-platform numbers come from MLC. Quantisation costs more answer quality than the bit count alone suggests. Scoring each quantised build’s exact match against the unquantised model’s own answers on SQuAD and Natural Questions, the paper’s figure puts 4-bit builds between about 50 and 61 percent, 6-bit between roughly 62 and 78 percent, and 2-bit and 3-bit builds as low as 15 percent. The authors note that 3-bit did not clearly beat 2-bit, and that 5-bit landed close to 4-bit while consuming more resources and degrading more than any 4-bit variant. Chart: Li et al., licensed CC BY 4.0. Exact match is measured against the answers of the same model before quantisation, not against ground truth. The 4-bit builds drew more power than the 3-bit ones rather than less. The paper’s table records 4-bit consuming more energy on all eight devices for the same 3B Llama model, rising from 10.13 to 13.05 mWh on the iPhone 15 Pro and from 14.54 to 14.68 mWh on the Pixel 7, which the authors put down to the higher-precision build occupying more GPU duty cycles. Surface temperature measured with a FLIR C5 thermal camera peaked at 75.4 °C on the Orange Pi 5 and 46.3 °C on the iPhone 15 Pro at 4-bit. Harmful output tracked the bit count more cleanly, with the hallucination rate on Llama-3-8B at 34.7 percent for 2-bit against 9.1 percent for 4-bit GPTQ and 7.9 percent at 8-bit. The authors name several limits on their own work. They cannot compare the two runtimes on iOS because llama.cpp does not support the iPhone, they profiled total memory only at 3-bit and 4-bit because higher-bit MLC builds exceeded the devices’ capacity, and they dropped models that would not fit at all, such as the unquantised Llama-2-7B at 13.11 GB. They also note that both runtimes expose few decoding options, with beam search unavailable in llama.cpp. Artificial Analysis has since run an independent benchmark of 33 quantised models on a single iPhone 17 Pro. Source: https://arxiv.org/abs/2410.05315 Read the article: https://llmobile.news/ticker/palmbench/BenchmarksQuantisationiPhoneAndroidMLC LLMhttps://llmobile.news/ticker/arm-generative-ai-on-mobile/https://llmobile.news/ticker/arm-generative-ai-on-mobile/Wed, 17 Jan 2024 15:00:00 +0100Arm showed a chat assistant running Meta’s Llama 2 7B on existing Android phones using three Cortex-A700 series CPU cores and LPDDR5x memory, at a text generation rate of 9.6 tokens per second. Gian Marco Iodice and Adnan AlSinan published the demo video and the measurement on January 17, 2024, and state that the video runs at actual speed and that everything runs locally on the device. Arm says the rate is faster than average human reading speed. Integer quantisation is what brought a 7B model into phone memory, according to Arm’s write-up. Arm puts the RAM needed for Llama 2 7B with 16-bit floating point weights at about 14 GB, and says quantising those weights to 4-bit shrinks the model four times, to roughly 4 GB. Arm warns that going down to 3-bit or 2-bit costs accuracy. Pinning each worker thread to a specific core with a thread affinity mask, rather than letting Android’s scheduler place it, improved latency by over 10 percent in Arm’s tests. Arm then wrote its own 4-bit matrix-by-vector and matrix-by-matrix routines, the two functions that dominate the work in a language model, tuned for Cortex-A700 series cores. The routines are built on two instructions the cores already carry, SDOT for dot products and SMMLA for 8-bit integer matrix multiply and accumulate. Arm reports that they cut the time to the first token by over 50 percent and raised text generation by 20 percent against the native implementation in llama.cpp, and said at the time that the routines would be available soon. Arm makes the case for the CPU on the shape of the workload rather than on peak throughput, arguing that the typical batch size and the balance of compute and bandwidth in language model inference suit a CPU, and that its programmability lets developers try new compression and quantisation schemes. The post names no NPU or GPU and offers no comparison against one. Arm puts the reach of the approach at around 99 percent of smartphones worldwide and says 70 percent of AI in third-party applications already runs on Arm CPUs. Arm later extended the same post with a second demo, a teaching assistant chatbot built on Microsoft’s Phi-3 3.8B, which Arm measures at just over 15 tokens per second on the same software optimisations. Arm names the Scalable Matrix Extension for Armv9-A as the next hardware step for these workloads. Source: https://developer.arm.com/community/arm-community-blogs/b/ai-blog/posts/generative-ai-on-mobile-on-arm-cpu Read the article: https://llmobile.news/ticker/arm-generative-ai-on-mobile/ArmAndroidllama.cppLlamaQuantisationhttps://llmobile.news/ticker/snapdragon-8-gen-3/https://llmobile.news/ticker/snapdragon-8-gen-3/Tue, 24 Oct 2023 19:00:00 +0200Qualcomm launched the Snapdragon 8 Gen 3 at its Snapdragon Summit in Hawaii on October 24, 2023, calling it its first mobile platform designed with generative AI in mind. On the product page the company states support for generative models with up to 10 billion parameters on device and up to 20 tokens per second for large language models. Qualcomm puts the Hexagon NPU at 98 percent faster than the previous generation and claims 40 percent better performance per watt for sustained AI inference. It lists the Kryo CPU at 30 percent more performance and 20 percent better power efficiency, and the Adreno GPU at 25 percent faster rendering with 25 percent better efficiency. Among the workloads Qualcomm names for local execution are Stable Diffusion, virtual assistants and chatbots. Devices were announced by ASUS, Honor, iQOO, MEIZU, Nubia, OnePlus, OPPO, realme, Redmi, RedMagic, Sony, vivo, Xiaomi and ZTE. Image: Qualcomm. Source: https://www.qualcomm.com/news/releases/2023/10/qualcomm-launches-premium-snapdragon-8-gen-3-to-bring-generative Read the article: https://llmobile.news/ticker/snapdragon-8-gen-3/QualcommChipsNPUAndroidhttps://llmobile.news/ticker/qualcomm-stable-diffusion-android/https://llmobile.news/ticker/qualcomm-stable-diffusion-android/Thu, 23 Feb 2023 17:00:00 +0100Qualcomm AI Research has demonstrated Stable Diffusion v1.5, a model with more than 1 billion parameters, running entirely on an Android phone powered by a Snapdragon 8 Gen 2. The company reports image generation in under 15 seconds for 20 inference steps at 512x512 pixels, a workload that until then ran in the cloud. Images: Qualcomm. Sample outputs from the demo, generated on the phone. To fit the model on the phone, Qualcomm quantised it from FP32 to INT8 using its AI Model Efficiency Toolkit and a post-training method called AdaRound, which adapts how individual weights are rounded. According to the company, the model needed no retraining after the conversion. Inference runs through the Qualcomm AI Engine direct framework on the Hexagon processor, with the model prepared in Qualcomm AI Studio. The company sums up the result in its post. This is the fastest inference on a smartphone and comparable to cloud latency. Source: https://www.qualcomm.com/news/onq/2023/02/worlds-first-on-device-demonstration-of-stable-diffusion-on-android Read the article: https://llmobile.news/ticker/qualcomm-stable-diffusion-android/QualcommImage generationQuantisationAndroidNPU