All updates

Google released Gemma 2 2B on July 31, 2024, an open model of 2.6B parameters that Google says produces outsized results by learning from larger models through distillation. Google reports it at 1126 Elo on the LMSYS Chatbot Arena leaderboard, captured on July 30, 2024, ahead of Mixtral 8x7B Instruct at 1114, GPT 3.5 Turbo 0314 at 1106 and Llama 2 70B chat at 1093, and states that the model surpasses all GPT-3.5 models there.

The technical report describes training the 2B size on 2 trillion tokens with knowledge distillation, where a bigger teacher model supplies a probability for every token in context and the small student is trained to match those probabilities rather than to predict the next token. In an ablation the authors ran, a 2B model trained over 500B tokens distilled from a 7B teacher averaged 67.7 across three benchmarks, against 60.3 for the same model trained from scratch. The released model holds 2.02B non-embedding parameters plus 590M embedding parameters, which come from the 256k-entry Gemini vocabulary, and it interleaves local sliding-window attention with global attention over an 8192-token context. Google DeepMind trained it on 512 TPUv5e chips.

Google names edge devices and laptops as deployment targets next to cloud serving on Vertex AI and Google Kubernetes Engine, and says the model is small enough for the free tier of T4 GPUs in Google Colab. For running it locally Google points to Ollama and to Gemma.cpp, its own lightweight C++ inference engine, and says support in MediaPipe, the on-device inference stack for Android, iOS and the web, is coming. Google adds that NVIDIA optimised the model with the TensorRT-LLM library for GeForce RTX cards and for Jetson modules at the edge.

Google put the weights on Kaggle, Hugging Face and Vertex AI Model Garden under the Gemma terms, which it calls commercially friendly for research and commercial applications, and made the model available to try in Google AI Studio. The same announcement added ShieldGemma, a set of safety classifiers built on Gemma 2 that filter hate speech, harassment, sexually explicit and dangerous content, and Gemma Scope, over 400 sparse autoencoders for inspecting the inner workings of Gemma 2 2B and 9B.

PyTorch released torchchat on July 30, 2024, a small codebase showing how to run Llama 3 and other language models on laptops, desktops and phones. For the phone path the team reports more than 8 tok/s for Llama 3 8B Instruct on a Samsung Galaxy S23 and on iPhone, using 4-bit GPTQ quantisation through ExecuTorch.

The repository names iOS 17 or later with at least 8 GB of RAM as its requirement, which it narrows to iPhone 15 Pro and newer or an iPad with Apple silicon, and on Android any device covered by the XNNPACK CPU backend, with 12 GB of RAM and 20 GB of storage recommended. Its model table marks Llama 3.2 1B and 3B, Llama 3.1 8B, Llama 2 7B, Mistral 7B, IBM Granite 3.1 and DeepSeek R1 Distill 8B as mobile friendly. The 11B vision version of Llama 3.2 and the 13B and 70B Llama 2 checkpoints carry no such mark.

Getting a model onto a phone is one export step that turns a downloaded checkpoint into a .pte file for ExecuTorch, the PyTorch edge runtime that reached version 1.0 in October 2025. The mobile preset shipped in the repository quantises the embedding table to 4-bit in groups of 32 values and the linear layers to 4-bit weights whose activations are quantised to 8-bit at runtime, in groups of 256. The quantisation guide offers 4-bit and 8-bit embeddings at group sizes from 32 to 256 across all execution modes, lists the dynamic-activation scheme as the supported one for ExecuTorch, and notes that smaller groups preserve accuracy while larger ones run faster.

Both demo apps are sample harnesses rather than shipped products. On iOS the repository points at the ExecuTorch LLaMA example project in Xcode, which needs a provisioning profile carrying Apple’s increased memory limit entitlement, after which the model and tokenizer files are copied into the app’s folder on the device. On Android the repository carries its own app skeleton for Android Studio, built against a prebuilt ExecuTorch library, with the model and tokenizer pushed to the phone over adb.

The other two paths in torchchat do not touch phones. One runs the model in Python through a chat and generate command line, a browser interface or a local server, the other compiles it ahead of time with PyTorch’s AOTInductor into a binary that a C++ runner loads on a desktop or server. The code is under a BSD 3-Clause license, and PyTorch halted active development on May 19, 2025, saying vLLM had become the dominant option for server-side inference.

Nolano AI, the University of Montreal and Mila published Spectra on July 17, 2024, a suite of 54 language models covering nine parameter counts from 99M to 3.9B, each trained on the same 300B tokens of SlimPajama. Every size exists as a half-precision model the authors call FloatLM, as GPTQ-quantised copies at 3, 4, 6 and 8 bits called QuantLMs, and as a TriLM, a model trained from scratch with every linear-layer weight restricted to -1, 0 or 1. The paper’s size table puts TriLM 3.9B at 10.76 billion bits, roughly 1.3 GB, against 63.83 billion bits for the half-precision model of the same parameter count, a factor of 5.9.

That is the comparison the suite is built for, since what fits on a device is bits rather than parameters. Averaged over six commonsense and reasoning benchmarks, the paper’s tables put TriLM 3.9B at 60.7 against 61.4 for FloatLM 3.9B, level with the 4-bit QuantLM 3.9B at 60.7 which needs 20.59 billion bits. Set against the models nearest to it in size, TriLM 3.9B beats the half-precision 830M model, 60.7 against 53.3 at 13.34 billion bits, and the 3-bit QuantLM 2.4B, 60.7 against 54.7 at 10.95 billion bits. On LAMBADA it reaches 61.6 against 61.1 for FloatLM 3.9B, and on TriviaQA 21.3 against 21.5.

During training a TriLM keeps floating-point latent weights and ternarises them on the fly in the forward pass, with one scale value per weight matrix shard and gradients passed through by a straight-through estimator, so only the ternary states and the scales are needed at inference. Embeddings and the output head stay in half precision across the suite. The authors position TriLM against BitNet b1.58, which also trains ternary weights from scratch, and say they normalise once before each linear layer instead of normalising, scaling and quantising activations to 8 bits four to seven times per transformer block; their own replication of the BitNet architecture at 1.1B, trained on 100B tokens, scores below TriLM 1.1B on the same benchmarks.

The paper names the weaknesses it measured. At 1.1B parameters the ternary model does not reach its half-precision counterpart, 51.6 against 54.9 on the six-benchmark average, and the 2.4B one still trails at 57.3 against 59.2, so the match with a half-precision model of the same parameter count holds at 3.9B only. On CrowsPairs, which measures how often a model prefers a stereotyped sentence, TriLM 3.9B lands at 65.4 percent against 64.7 for FloatLM 3.9B and 62.6 for the smaller FloatLM 830M, so the ternary model stereotypes more than the half-precision model closest to it in bits. No model in the suite beats random guessing on TruthfulQA, and while TriLM 3.9B holds up on Penn Treebank and LAMBADA perplexity, it stays behind FloatLM 3.9B on web corpora including Dolma, C4 and RefinedWeb, which the authors read as weaker memorisation of training data.

The weights and more than 500 intermediate checkpoints are on Hugging Face under Apache 2.0. The TriLM checkpoints are published unpacked to FP16, which the model cards say makes them compatible with ordinary half-precision matrix multiplication and leaves the released files at full half-precision size. The authors list smartphones, laptops and automobiles among the resource-constrained devices they see for ternary models, and end by calling for work on the toxicity, stereotyping and web-corpus gaps they measured.

Hugging Face released SmolLM on July 16, 2024, a family of three base models at 135M, 360M and 1.7B parameters. The team says the models are designed to be small enough to run locally across a range of hardware, from smartphones to laptops, and cites the iPhone 15 with 6 GB of DRAM and the iPhone 15 Pro with 8 GB as reference points. Hugging Face trained the 135M and 360M models on 600B tokens each and the 1.7B model on 1T tokens.

The substance is the training data, released as SmolLM-Corpus. Cosmopedia v2 contributes 28B tokens of synthetic textbooks, stories and articles generated by Mixtral-8x7B-Instruct, 39 million documents built from a predefined list of 34,000 topics taken from the BISAC book classification instead of from unsupervised clustering. FineWeb-Edu contributes 220B deduplicated tokens of educational web pages, pulled out of FineWeb by a quality classifier trained on annotations from Llama3-70B-Instruct. Python-Edu applies the same idea to code, keeping 4B tokens of the 40B Python tokens in The Stack that scored 4 or higher on an educational code classifier.

On Hugging Face’s own evaluation, run with the same lighteval setup for every model it could download, SmolLM-135M averages 39.50 across MMLU, TriviaQA, ARC, PIQA, HellaSwag, OpenBookQA and Winogrande, against 32.75 for GPT2-137M and 32.49 for Pythia-160M. SmolLM-360M averages 44.44 against 39.60 for Qwen2-500M and 39.32 for Qwen1.5-500M, and SmolLM-1.7B averages 51.97 against 50.36 for Microsoft’s Phi-1.5 and 47.98 for Qwen2-1.5B. Meta’s MobileLLM is not publicly available, so Hugging Face took its per-benchmark numbers from the paper and reports no average for those rows.

Hugging Face also benchmarked the memory footprint of each size. SmolLM-135M takes 269.03 MB in bf16, 162.87 MB at int8 and 109.78 MB at int4. SmolLM-360M takes 723.65 MB, 409.07 MB and 251.79 MB across the same three precisions, and SmolLM-1.7B takes 3422.76 MB, 1812.14 MB and 1006.84 MB.

The weights are on Hugging Face under Apache-2.0, with ONNX checkpoints published beside the transformers ones and a GGUF build for llama.cpp announced as planned. Hugging Face also put up browser demos of the instruction-tuned 135M and 360M models that run client-side through WebGPU, with no server call. The corpus is released separately under ODC-By.

Alibaba’s Qwen Team published the Qwen2 Technical Report on July 15, 2024, covering five models from 0.5B to 72B parameters. The report states that the two smallest, Qwen2-0.5B and Qwen2-1.5B, are designed for easy deployment on portable devices such as smartphones, earphones and smart glasses, while the larger sizes target GPUs. Qwen2-1.5B scores 56.5 on MMLU, a multiple-choice knowledge test, against 52.7 for Microsoft’s Phi-2, which the same table credits with about twice the non-embedding parameters.

Both small sizes tie the input embedding to the output layer, so a single weight matrix does both jobs and the shared vocabulary of 151,646 tokens is stored once rather than twice. The report puts non-embedding parameters at 0.3B for Qwen2-0.5B and 1.2B for Qwen2-1.5B. All five models use grouped query attention, where several query heads share one set of key and value heads, with 14 query heads over 2 key-value heads at 0.5B and 12 over 2 at 1.5B, which the authors say lowers the key-value cache size per token against Qwen1.5 and with it the memory footprint during long-context inference.

Alibaba trained Qwen2-0.5B on a 12 trillion token dataset and the other dense models on a 7 trillion token one, reporting that the larger corpus did not pay off at the bigger sizes. For the instruction-tuned versions the report gives Qwen2-1.5B-Instruct 52.4 on MMLU, 61.6 on the GSM8K grade-school maths set and 29.0 on IFEval, which checks whether a model obeys explicit formatting instructions, against 37.9, 40.1 and 20.0 for Qwen2-0.5B-Instruct. The authors attribute the gain over the similarly sized Qwen1.5 models mainly to that data scaling.

Context length differs across the family. The authors tested retrieval from long documents with Needle in a Haystack and report that the two small models handle 32K tokens, Qwen2-57B-A14B-Instruct 64K, and the 7B and 72B instruction-tuned models 128K with the YARN extension applied. The pre-training data covers approximately 30 languages, among them English, Chinese, Spanish, French, German, Arabic, Russian, Korean, Japanese, Thai and Vietnamese.

Alibaba put the weights on Hugging Face and ModelScope. The Qwen2 blog post of June 7, 2024 states that Qwen2-0.5B, 1.5B, 7B and 57B-A14B carry the Apache 2.0 license while Qwen2-72B and its instruction-tuned variant stay on Alibaba’s own Qianwen License, and it names MLX, llama.cpp, Ollama and LM Studio among the projects that run the models locally.

Researchers at Peking University and the Beijing University of Posts and Telecommunications published llm.npu on July 8, 2024, an inference engine that runs the prefill stage of a language model on a phone’s NPU. They tested it on a Redmi K70 Pro with a Snapdragon 8 Gen 3 and a Redmi K60 Pro with a Snapdragon 8 Gen 2, using Qwen1.5-1.8B, Gemma-2B, Phi-2 2.7B, LLaMA2-7B and Mistral-7B. On 1024-token prompts on the K70 Pro the authors report prefill 7.3x to 38.4x faster than the CPU engines MNN and llama.cpp and 1.27x to 43.6x faster than the GPU engines TFLite and MLC-LLM, and for Qwen1.5-1.8B they measure 1,106 tok/s against 151 tok/s for MNN on the CPU and 57 tok/s for llama.cpp on the CPU. Energy was measured on the K60 Pro, the only device they could root, where prefill used 1.85x to 59.5x less energy than the same baselines.

Prefill is the stage where the model reads the whole prompt before it writes its first token, and the paper puts it at 88.3% to 98.8% of total latency on a mobile CPU for UI automation, chat summaries and context-aware generation, and at 54.2% to 91.7% on a mobile GPU. A mobile NPU is fast at integer matrix multiplication but expects fixed tensor shapes, while prompts arrive at any length, so every new length means rebuilding the graph. Quantised activations also contain outliers, single values far outside the usual range, and the accurate remedy is per-group quantisation, which mobile NPUs cannot execute directly and which the authors measure at up to 10.7x overhead when it is split into sub-tensor multiplications.

llm.npu answers both by cutting a prompt into fixed 256-token chunks that run on pre-built graphs, sharing the operators that do not depend on prompt length so the memory cost drops by up to 4x, pulling the outlier values out to the CPU or GPU to run in parallel, and scheduling transformer blocks across the processors out of order. Accuracy stays within 1% of FP16 on LAMBADA, HellaSwag, WinoGrande, OpenBookQA and MMLU. Across three application workloads, UI automation, context-aware email replies and chat summaries, the authors measure end-to-end latency 1.4x to 32.8x lower than the baselines, with decoding still running unoptimised on the CPU.

The system is built on the group’s own mllm inference framework and Qualcomm’s QNN SDK, in about 10,000 lines of C, C++ and assembly, and the code is public in the mllm repository under an MIT license. The paper appeared at ASPLOS 2025 in Rotterdam in spring 2025. PowerInfer-v2, the one comparison system that also uses the NPU for prefill, is not open source, so the authors took its figures from its paper and report a 3.28x to 5.6x speedup over it.

Five Apple researchers presented MobileCLIP at CVPR 2024 on June 20, a family of image-text models built for phones. They exported each model with Core ML Tools 7.0 and timed it at batch size 1 on an iPhone 12 Pro Max running iOS 17.0.3, and their Table 7 gives the smallest variant, MobileCLIP-S0, 1.5 ms for the image encoder and 1.6 ms for the text encoder at 67.8% zero-shot accuracy on ImageNet. The same authors measure OpenAI’s ViT-B/16 CLIP on that phone at 11.5 ms and 3.3 ms for 68.3%, and both models average 58.1 across the 38 datasets of the DataComp evaluation suite.

Multi-modal reinforced training, the method the paper introduces, moves the expensive part of distillation out of the training loop and into the dataset. Apple reinforces the DataComp dataset once, writing five synthetic captions per image with the CoCa model in OpenCLIP, storing the parameters of strong random image augmentations, 30 per image for the 12M subset and 10 for the billion-sample set, and storing the embeddings that two ViT-L/14 teacher models produce for those augmented images and for the real and synthetic captions. Training afterwards reads the captions and the teacher embeddings from disk, so the captioning model and the teachers never run again however many models are trained on the set.

The paper puts numbers on what that saves. One epoch over the 12.8M samples of the reinforced set on a single node of 8 A100-80GB GPUs takes 1.3 hours, the same as an epoch on the unreinforced data, against 21.1 hours when the captioning model and the teacher ensemble run during training, according to Table 4d. The cost moves to storage, and Table 4c puts DataComp-1B at 90 TB against 140 TB for the reinforced DataCompDR-1B, with the 12M subset going from 0.9 TB to 1.9 TB.

MobileCLIP comes in four sizes, and Table 7 lists image and text encoder parameters separately. S0 pairs an 11.4M-parameter image encoder with a 42.4M-parameter text encoder, S1 uses 21.5M and 63.4M at 2.5 ms and 3.3 ms, S2 uses 35.7M and 63.4M at 3.6 ms and 3.3 ms, and MobileCLIP-B keeps a standard ViT-B/16 image encoder at 86.3M and 10.4 ms. Zero-shot ImageNet accuracy across the four runs 67.8%, 72.6%, 74.4% and 76.8%, and a version of B trained on 39B seen samples instead of 13B reaches 77.2%. All three small variants replace the vision transformer with a convolution-transformer hybrid derived from Apple’s FastViT, and S0 also swaps the text encoder for one that mixes 1-D convolutions with four self-attention layers, which the authors report as 5% smaller and 15.8% faster than the 12-layer transformer text encoder that S1 and S2 keep.

Apple released the code, the weights and the reinforced data at apple/ml-mobileclip, with checkpoints for all four variants and the longer-trained B, and with DataCompDR-1B and DataCompDR-12M on Hugging Face as synthetic captions, augmentation parameters and teacher embeddings keyed to DataComp images rather than as the images themselves. The repository lists the code under MIT, the models under Apple’s own machine learning research model terms, and the data under CC BY-NC-ND. Apple showed zero-shot scene classification running in real time on an iPhone in the CVPR exhibit hall, and added an iOS app for the same task to the repository in November 2024.

TensorOpera released Fox-1 on June 13, 2024, a decoder-only language model with 1.6B parameters trained on 3 trillion tokens of text and code. The company says the run followed a three-stage data curriculum at 8K sequence length, without saying what each of the three stages contained.

Fox-1 uses grouped query attention, where several attention heads share one set of key and value vectors to cut memory traffic, with 16 attention heads over 4 key-value heads. TensorOpera describes the model as deeper than others of its size, 78% deeper than Gemma-2B, 33% deeper than Qwen1.5-1.8B and StableLM-2-1.6B, and 15% deeper than Apple’s OpenELM-1.1B.

TensorOpera reports an average of 47.13% across ARC Challenge, HellaSwag, MMLU, TruthfulQA, Winogrande and GSM8k, which it ran in the Open LLM Leaderboard setup on one machine with 8 H100 GPUs. It puts Qwen1.5-1.8B at 46.81%, Gemma-2B at 46.36%, StableLM-2-1.6B at 45.92% and OpenELM-1.1B at 38.28%. Fox-1 leads that group on GSM8k with 36.39% against 34.04% for Qwen1.5-1.8B and 17.06% for Gemma-2B, and trails Qwen1.5-1.8B on MMLU with 43.05% against 47.15%.

The device claim in the announcement is a target rather than a measurement. TensorOpera puts the model’s FP16 memory use at 3,703 MiB, against 5,379 MiB for Gemma-2B, 4,739 MiB for Qwen1.5-1.8B and 3,852 MiB for StableLM-2-1.6B, and says that makes Fox-1 comparable for on-device deployment. The stated plan is that developers train on the company’s cloud platform and then deploy, personalise and monitor models on smartphones and AI-enabled PCs through its FedML platform. No figure for speed or latency on a phone appears in the post, and the throughput number it does give, over 200 output tokens per second per user, was measured with vLLM on a single H100.

TensorOpera published the base model under the Apache 2.0 license on Hugging Face and on its own platform, and said an instruction-tuned version would follow.

Five researchers at Beijing University of Posts and Telecommunications measured language model inference on four Android phones and presented the results at the EdgeFM workshop of MobiSys 2024, deploying 22 models from 0.5B to 7B parameters through llama.cpp. For the 18 models they charted at 4-bit quantisation, accuracy for most of them sat between 55 and 60 percent, averaged across Winogrande, HellaSwag and MMLU, and latency came out at about 200 ms per token averaged over the four devices. The phones were a Xiaomi 14 with a Snapdragon 8 Gen 3 and 16 GB of RAM, a Pixel 7 Pro with a Tensor G2 and 12 GB, a Xiaomi 12S with a Snapdragon 8 Gen 1+ and 12 GB, and a Meizu 18 Pro with a Snapdragon 888 and 8 GB.

Most of the 7B models needed about 4 GB of memory, which the authors put at half or a third of a current phone’s RAM. Their memory leaderboard has Bloom-7B highest, at 1.27 times the footprint of MPT-7B at the same parameter count, and Qwen1.5 0.5B lowest at under 1 GB. Memory use did not move with input or output length in their runs, with Baichuan holding at 4.83 GB whether the prompt was 10 or 50 tokens long and whether 1 or 1000 tokens were generated, which they trace to llama.cpp allocating one large pool at startup rather than sizing it per phase.

Newer chips helped the two inference phases unevenly. Against the Snapdragon 888, the authors measured the Snapdragon 8 Gen 3 raising prefill throughput on LLaMA 2 by 81% but decode throughput by only 43%, with the Snapdragon 8 Gen 1+ at 45% and 34%. Decode is the phase that emits one token at a time and so governs how fast text appears on screen, and there the gap between the 8 Gen 3 and the 8 Gen 1+ was 7%. Continuous prefill runs also pushed the Xiaomi 12S, Xiaomi 14 and Meizu 18 Pro into thermal throttling, with per-token times drifting during a test series.

The processor comparison went against the GPU in most cases. In the engine test llama.cpp on the 8 Gen 3 CPU prefilled 1.63 times faster than MLC LLM on the same phone’s GPU, 10.3 tokens per second against 6.3, while MLC on the GPU decoded faster at 9.8 against 6.6. On the mid-range Snapdragon 7 Gen 2 the CPU was ahead of the GPU in both phases, 4.59 and 4.65 against 3.3 and 3.5. The authors put this down to mobile GPUs being built for rendering rather than general-purpose compute, and to the extra cost of moving data.

Quantisation cost less than the hardware choices did. Testing eight bit-widths on LLaMA 2, the authors report 4-bit as the best compromise, giving up 0.6% average accuracy against the 32-bit model while cutting model size 8 times, where 2-bit halves the size again for four times the accuracy loss. Code and results are published as TinyLLMLeaderBoard on GitHub, and the six-page paper is free to read in the ACM Digital Library.

Apple introduced Apple Intelligence on June 10, 2024, built on a roughly 3-billion-parameter language model that runs on the device and a larger model that runs on Apple silicon servers through Private Cloud Compute. The on-device model uses a 49K token vocabulary, the server model 100K.

Apple compresses the on-device model with a mix of 2-bit and 4-bit weights, averaging 3.7 bits per weight, and says an option at 3.5 bits per weight costs no significant quality. On an iPhone 15 Pro the company reports a time to first token of about 0.6 milliseconds per prompt token and 30 tokens per second, both measured before token speculation.

Rather than ship a separate model per feature, Apple swaps LoRA adapters on top of the shared base model. The adapters are stored in 16 bits and, at rank 16, take tens of megabytes each. Apple says human graders preferred its models over comparable ones including GPT-3.5, GPT-4, Llama, Mistral and Gemma across its evaluations.

Researchers at the IPADS lab of Shanghai Jiao Tong University published PowerInfer-2 on June 10, 2024, an inference framework for running language models that exceed a phone’s memory. On a OnePlus 12, a Snapdragon 8 Gen 3 device with 24 GB of DRAM of which 19 GB can be occupied by an app, the authors measure TurboSparse-Mixtral-47B decoding at 11.68 tokens per second, and describe it as the first system to serve a 47B model on a smartphone. In the same configuration they report 3.12 times the speed of LLMFlash and 21.2 times the speed of llama.cpp. LLMFlash here is their own reimplementation of Apple’s LLM in a flash techniques inside llama.cpp, which they built because that work was never released as code.

PowerInfer-2 splits each feedforward layer into neuron clusters, small groups of neurons that tend to activate together, and treats a cluster rather than a whole matrix as the unit of work. Frequently activated clusters go to the phone’s NPU for dense arithmetic, while rarely activated ones go to the CPU, which copes better with irregular sparse work. Weights that are not already in an in-memory cache are read from UFS flash on demand, and those reads are overlapped with computation so that a core can work on one cluster while the next is still arriving. For prompt processing the system loads whole layers with sequential reads, which the authors measure at 3 times the throughput of random reads, and prefetches the next layer while the current one computes.

The authors vary how much memory the app may use from 7 GB to 19 GB with the 47B model. At 7 GB the cache holds only 1.8 percent of the feedforward weights, so nearly every neuron has to be fetched from flash for every token and decoding falls to 2.13 tokens per second, which is still 1.84 times LLMFlash. The 47B model activates only about 3B parameters per token thanks to its mixture-of-experts structure and added sparsity, which is why it decodes at roughly the pace of a 7B model in the same test. Streaming weights off storage to fit a large model into a small budget is the same bet Edge0 makes at the level of whole experts rather than individual neurons.

On OpenCompass at 4-bit weights, the authors put PowerInfer-2 at an average of 78.38 across Arc-Challenge, Arc-Easy, MMLU and GSM8K for Qwen2 7B against 79.25 for llama.cpp, and at 68.35 against 70.12 for Bamboo-7B, with Qualcomm’s QNN engine at 56.93 and 63.26. Token latency is uneven, and for the 47B model with half the feedforward weights offloaded they report a mean of 99.76 ms per token and a 99th percentile of 140.56 ms, tracking a cache miss rate that averages 3.5 percent but reaches 18.9 percent at the 99th percentile. Prompt processing for Bamboo-7B runs at 404.6 tokens per second with half the weights offloaded, against over 700 tokens per second with everything in memory. They measure energy at 0.257 joules per token against 0.373 for QNN and 0.672 for llama.cpp.

PowerInfer-2 uses privileged system calls such as memory locking, so the authors built and ran it on rooted Android phones, and they state that nothing in the design requires kernel changes and that it should port to iOS. It also needs models with predictable sparsity, which is why the group trained sparsified TurboSparse versions of Mistral and Mixtral, and on an unmodified SiLU-based Mistral-7B the gain over LLMFlash drops to 2.4 times against 4.6 times for the ReLU-based Bamboo-7B. The system is 12,000 lines of C++ added to the group’s PowerInfer engine for PCs, which is MIT-licensed, but the announcement there links the paper and the model weights rather than a release of the PowerInfer-2 code. The paper was last revised in December 2024.

Researchers at Beijing University of Posts and Telecommunications proposed that a phone ship one shared multimodal model instead of letting every app bundle its own, in a paper published in the ACM MobiCom 2024 proceedings on May 29, 2024. The operating system and the hardware co-manage that model like firmware, unchangeable by apps or by the OS itself, exposed to applications as a system service, and each app reaches it through a small adapter fine-tuned offline for its own task. Their prototype, called M4, holds 9.2B parameters and needs 7.5 GB of peak memory, and the authors report it reaching accuracy comparable to purpose-built models on 85% of the 50 datasets in a benchmark they assembled from 38 mobile AI tasks across five input types.

What the shared model replaces is one small model per app per task. The paper’s baselines are 50 task-specific models of 1M to 500M parameters each, one per dataset, against which M4’s adapters run from 1,000 to 10 million parameters, so each added task costs under 10 MB. Measured on an Nvidia Jetson Orin NX, 4-bit M4 needs 6.1 GB of storage to serve all 50 tasks against 15.2 GB for the 50 separate models, with the crossover at about 15 tasks, and 7.5 GB of peak memory against roughly five times that. The authors state that on a device with 12 GB of memory the 4-bit model plus all 50 adapters fits, where only 20 of the 50 task-specific models would.

The prototype is slower than the models it replaces. On the Jetson Orin NX with 16 GB, the authors measured M4 averaging 18 times the inference latency of the task-specific models across the 50 tasks and 19 times the energy, 3.6 s against 0.2 s. On a Pixel 7 Pro CPU they measured an average of 6.8 s against 0.54 s, and their per-task breakdown puts image classification at 2.10 s and question answering at 6.34 s to the first token and 0.24 s per token after it. They state that M4 cannot currently run on a stock smartphone GPU or NPU at all, because those processors lack support for the operators it uses.

The NPU numbers in the paper are a projection rather than a measurement. The authors estimate that M4 on an NPU would average 0.48 s and 1.3 J, under the 0.54 s and 2.9 J they measured for task-specific models on the Pixel 7 Pro CPU, but they derive that by applying the CPU-to-NPU ratio they observed for task-specific models, not by running M4 on an NPU. Their case for a simpler accelerator rests on a separate Pixel 7 Pro measurement, where they converted 110 downloaded models to TensorFlow Lite and only 8% ran entirely on the NPU, those gaining a median speedup above 20 times over the CPU. M4 itself uses 39 operator types against the 156 that the 50 task-specific models need between them.

The authors name their own limits. They write that the accuracy results come from an A100 and the Jetson board rather than from phones, that M4 underperforms task-specific models on some tasks including translation, and that a prototype assembled from off-the-shelf pre-trained models is “still highly inefficient in terms of accuracy and model parameter size”. Its backbone is Meta’s LLaMA-7B at 8-bit, with encoders taken from ImageBind and Whisper, and they note that adapters trained against one backbone stop working when the backbone is upgraded, so the design still needs a stable interface between the two. Code and benchmark are published at github.com/UbiquitousLearning/MobileFM, and the paper carries ACM copyright rather than an open license.