https://llmobile.news/tags/arm/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.news/ticker/flexserve/https://llmobile.news/ticker/flexserve/Tue, 10 Mar 2026 01:31:25 +0100Researchers at Shanghai Jiao Tong University published FlexServe on March 10, 2026, a serving system that keeps an on-device language model’s weights and the text going through it inside ARM TrustZone’s secure world. Running Llama 3.2 3B, Llama 3.1 8B, Qwen3 1.7B and Qwen3 8B quantised to 8-bit on a NanoPC-T6 board with a Rockchip RK3588, 16 GB of memory and a 6 TOPS NPU, the authors measure time to first token 4.41 percent above the same inference run without any protection, averaged over their prompt lengths. Against a TrustZone baseline that already uses a secure NPU and a loading pipeline, they report FlexServe 1.43 to 2.42 times faster, 1.85 times on average. TrustZone splits the chip into a normal world, where Android and every app runs, and a secure world the normal world cannot read. FlexServe puts the inference engine, the weights, the prompt, the generated text and the KV cache on the secure side, so a kernel an attacker has taken over cannot read any of it out of memory, and the authors justify that assumption with Linux’s 40 million lines of code and 9756 CVEs. Both halves are worth stealing in their account, since a vendor’s fine-tuned weights cost real money to train yet sit on a stranger’s device, and the apps calling a shared model feed it chat history and screen contents. The authors name the boundary they cannot cross, which is that an app in the normal world still receives its own input and output there, so a compromised kernel can take those, and only the weights stay protected in that case. Plain TrustZone is slow at this because it hands out secure memory as a few large contiguous regions. Reserving 8 GB at runtime therefore means the kernel’s contiguous allocator compacting scattered pages, which the authors time at 6440.67 ms against 568.58 ms for FlexServe’s page-granular secure memory, and the secure world cannot normally touch the NPU at all, leaving 30.06 s of CPU computation where the NPU needs 1.94 s. FlexServe carves secure pages out with the stage-2 page tables that ARM’s virtualisation extension already provides, and flips the NPU between protected and unprotected mode in 0.21 ms. Handing 8 GB back drops from 732.85 ms to 80.50 ms, because the pages are released lazily rather than zeroed in one pass. A scheduler decides which models stay resident when several share the device, holding the earliest layers of each in secure memory so the pipeline does not stall and prefetching the next model while the current one is still generating. With a 4 GB cache and requests drawn at random from UltraChat, OpenAssistant, Dolly and Alpaca, the authors measure pairs of models answering 8.84 times faster than the plain secure-world baseline and 2.53 times faster than the optimised one. Across ten agent workflows, where one model’s finished answer feeds the next, the gains reach 24.30 times and 4.05 times, averaging 14.15 and 2.94. Serving one phone-side model to several callers is also what ElastiLM works on, by resizing the shared model to each request’s latency budget rather than isolating it. The protection costs the rest of the device something as well. A SQLite instance holding 3 GB in the normal world runs 1.31 times slower while FlexServe holds 8 GB in the background, against 3.27 times slower with the contiguous allocator, and the authors put the standing virtualisation cost at 2.46 percent on SPEC CPU, which they switch off when no secure inference is running. They list the prototype’s limits themselves, a development board rather than a phone, Llama 3 and Qwen3 series models only, no GPU in the pipeline, and an NPU runtime slow enough that computation becomes the bottleneck, so a faster NPU would widen their lead. Side-channel attacks, physical attacks and denial of service are out of scope, and the paper names no code release. Source: https://arxiv.org/abs/2603.09046 Read the article: https://llmobile.news/ticker/flexserve/ResearchArmNPUMemoryLlamahttps://llmobile.news/ticker/arm-generative-ai-on-mobile/https://llmobile.news/ticker/arm-generative-ai-on-mobile/Wed, 17 Jan 2024 15:00:00 +0100Arm showed a chat assistant running Meta’s Llama 2 7B on existing Android phones using three Cortex-A700 series CPU cores and LPDDR5x memory, at a text generation rate of 9.6 tokens per second. Gian Marco Iodice and Adnan AlSinan published the demo video and the measurement on January 17, 2024, and state that the video runs at actual speed and that everything runs locally on the device. Arm says the rate is faster than average human reading speed. Integer quantisation is what brought a 7B model into phone memory, according to Arm’s write-up. Arm puts the RAM needed for Llama 2 7B with 16-bit floating point weights at about 14 GB, and says quantising those weights to 4-bit shrinks the model four times, to roughly 4 GB. Arm warns that going down to 3-bit or 2-bit costs accuracy. Pinning each worker thread to a specific core with a thread affinity mask, rather than letting Android’s scheduler place it, improved latency by over 10 percent in Arm’s tests. Arm then wrote its own 4-bit matrix-by-vector and matrix-by-matrix routines, the two functions that dominate the work in a language model, tuned for Cortex-A700 series cores. The routines are built on two instructions the cores already carry, SDOT for dot products and SMMLA for 8-bit integer matrix multiply and accumulate. Arm reports that they cut the time to the first token by over 50 percent and raised text generation by 20 percent against the native implementation in llama.cpp, and said at the time that the routines would be available soon. Arm makes the case for the CPU on the shape of the workload rather than on peak throughput, arguing that the typical batch size and the balance of compute and bandwidth in language model inference suit a CPU, and that its programmability lets developers try new compression and quantisation schemes. The post names no NPU or GPU and offers no comparison against one. Arm puts the reach of the approach at around 99 percent of smartphones worldwide and says 70 percent of AI in third-party applications already runs on Arm CPUs. Arm later extended the same post with a second demo, a teaching assistant chatbot built on Microsoft’s Phi-3 3.8B, which Arm measures at just over 15 tokens per second on the same software optimisations. Arm names the Scalable Matrix Extension for Armv9-A as the next hardware step for these workloads. Source: https://developer.arm.com/community/arm-community-blogs/b/ai-blog/posts/generative-ai-on-mobile-on-arm-cpu Read the article: https://llmobile.news/ticker/arm-generative-ai-on-mobile/ArmAndroidllama.cppLlamaQuantisation