All updates

Prism ML has released 1-bit Bonsai 8B, a language model with 8.2 billion parameters that stores every weight in a single bit. Embeddings, attention layers, MLP layers and the LM head are all 1-bit, which brings the model down to 1.15 GB. The company also released 4B and 1.7B versions of the family.

On Apple hardware Prism reports about 40 tok/s on an iPhone 17 Pro and about 44 tok/s on an iPhone 17 Pro Max, against 131 tok/s on an M4 Pro Mac. An NVIDIA RTX 4090 reaches 368 tok/s, compared with 59 tok/s for a 16-bit 8B model in Prism’s chart. The 16-bit model reaches 16 tok/s on the M4 Pro and does not fit on the iPhone 17 Pro Max.

Prism measured 0.074 mWh per token on the M4 Pro and 0.068 mWh per token on the iPhone 17 Pro Max, which it puts at 4 to 5 times less than 16-bit counterparts. On the RTX 4090 the figures are 0.276 mWh for 1-bit Bonsai 8B and 1.134 mWh for the 16-bit model.

The company gives the 8B model an average benchmark score of 70.5, made up of 65.7 on MMLU Redux, 50.0 on MuSR, 88.0 on GSM8K, 73.8 on HumanEval+, 79.8 on IFEval and 65.7 on BFCLv3. It compares the model with 8B-class models such as Qwen3 8B, Llama 3.1 8B, Ministral3 8B and LFM2 8B. On its own “intelligence density” metric, a score per GB of model size, Prism lists 1.06 for Bonsai 8B and 0.10 for Qwen3 8B.

The models are released under the Apache 2.0 license. They run on Apple devices via MLX and on NVIDIA GPUs via llama.cpp CUDA, and the iOS app Locally AI can run them on iPhone. Weights are in the Hugging Face collection, and the demo repository holds a whitepaper and quick start instructions, along with a Colab notebook.

Four authors at Conscious Engines published a benchmark of Qwen 2.5 1.5B at 4-bit across four edge platforms on March 24, 2026, feeding each device the same 258-token prompt 20 times in a row with a 1-second gap instead of once. On an iPhone 16 Pro the first two runs peak at 40.49 tokens per second, and from run 17 onwards the phone holds a plateau of 23.67 tok/s, which the authors put at 41.5 percent below that peak.

The iPhone passes through the three thermal states iOS reports to an app. Runs 1 and 2 sit in Normal and average 36.83 tok/s, runs 3 to 16 sit in Warm and average 24.96 tok/s, and runs 17 to 20 sit in Hot at 23.67 tok/s with a variation of 1.2 percent. The authors write that the 1-second gap allows no thermal recovery, and they measure 5 percent of battery drained over the 20 runs, which they project to roughly 400 replies of this length per charge.

A Samsung Galaxy S24 Ultra degrades more gently in the same test. Its Adreno 750 GPU boosts to 1,000 MHz on runs 4 and 5 for a peak of 12.21 tok/s, then steps down to a 720 to 770 MHz plateau from run 8 where the authors measure 10.38 tok/s, 15 percent off the peak, with a low of 9.55 tok/s on the final run. They stress that reaching this plateau took deliberate tuning, with the display off, the context capped at 2,048 tokens and MLC-LLM’s prefill chunk cut to 128 tokens, and that earlier unconstrained runs hit an OS-enforced GPU frequency floor.

Neither phone ran the model on its NPU. The paper states that MLX dispatches to the A18 Pro GPU through Metal and does not target Apple’s Neural Engine, and that the S24 Ultra ran through MLC-LLM on the Adreno GPU, leaving the Snapdragon 8 Gen 3’s Hexagon NPU outside the measurement. The one NPU that did run the model, a Hailo-10H module in a Raspberry Pi 5, was also the only part that never throttled, holding 6.914 tok/s across all 20 runs with a variation of 0.04 percent at under 2 W, its CPU at 52.7°C and the NPU at 58.5°C, with attention layers that hailo-ollama does not offload falling back to the Pi’s Cortex-A76 CPU. A user running llama.cpp’s Hexagon NPU backend on a Snapdragon 8 Gen 3 phone reported 12.5 tok/s of generation on Gemma 3 4B and said the phone stayed cool.

A laptop RTX 4050 on battery led on speed at 131.7 tok/s and 34.12 W, its GPU climbing from 55°C to 70°C without throttling. Energy per token comes out at 270.5 mJ for the Hailo module measured across the whole system, 297.3 mJ for the RTX 4050 measured at the GPU, and 143.0 mJ for the S24 Ultra plateau derived from the Android fuel gauge with the display off. The authors state that the three readings differ in scope and call the resemblance suggestive rather than settled, and report no energy figure for the iPhone because iOS exposes no per-component power reading. They tested one model, one prompt and one unit per platform across four different runtimes, and describe the results as characterising hardware and software together rather than hardware on its own.

Meta AI published MobileLLM-Flash on March 16, 2026, three models of 350M, 650M and 1.4B parameters whose shapes were picked by measuring how fast each candidate actually ran on a phone. Meta reports the family reaching up to 1.8 times faster prefill and 1.6 times faster decode than Liquid AI’s LFM2 models on mobile CPUs, at comparable or higher accuracy, with support for up to 8k tokens of context.

Meta’s first MobileLLM paper argued in 2024 that deep and thin networks win below a billion parameters, and the Flash paper states that such deep-and-thin structures often fail to improve on-device latency. Meta exported 100 architectures to a phone and found parameter count a poor predictor of measured prefill and decode speed, at a Kendall rank correlation of 0.40, with FLOPs little better at 0.46 and 0.55. The 1.4B model therefore runs 16 layers at a width of 2048, against the 30 layers at 1280 of MobileLLM-Pro-1B.

Meta ran the latency tests on a Samsung Galaxy S25, a phone with a Snapdragon 8 Elite chipset, an octa-core CPU and 12 GB of memory, exporting each model through ExecuTorch to the XNNPACK CPU backend at 4-bit weights, 8-bit dynamic activations and a quantised KV cache, on 4 CPU threads and averaged over three runs after a warmup. At a 2k context the 650M model reaches its first token in 3.34 s against 6.01 s for LFM2 700M, and decodes at 85.35 tok/s against 53.57 tok/s. Meta repeated the 1k-context test on an iPhone 17, where the 650M model takes 2.81 s to the first token against 3.40 s for LFM2 700M.

Most of the speed comes from letting some layers skip attention, so a token passes through those blocks without being compared to every earlier token. The 350M model keeps full attention in 7 of its 12 layers and the 650M model in 8 of 13, while the 1.4B model attends in all 16. Meta reports that skipping consistently beat sliding window attention, which instead limits each token to a fixed span of recent ones, and that more than three skipped layers in a row costs accuracy at the same latency, dropping TriviaQA exact match from 33.2% to 8.8%.

Meta built the three models by pruning a shallow 1.8B variant of MobileLLM-Pro and inheriting its weights, which the paper says took about 35% of the tokens that pretraining MobileLLM-Pro required. Across nine reasoning and knowledge tasks Meta reports averages of 45.46, 48.57 and 55.06 for the three sizes, against 44.92, 48.52 and 50.48 for LFM2 at 350M, 700M and 1.2B, 51.79 for Gemma 3 1B and 49.01 for Llama 3.2 1B. After instruction tuning, the 1.4B model scores 47.89 on MMLU and 46.34 on HumanEval, against 49.30 and 37.80 for Llama 3.2 1B. The paper announces no release of the weights.

Researchers at Shanghai Jiao Tong University published FlexServe on March 10, 2026, a serving system that keeps an on-device language model’s weights and the text going through it inside ARM TrustZone’s secure world. Running Llama 3.2 3B, Llama 3.1 8B, Qwen3 1.7B and Qwen3 8B quantised to 8-bit on a NanoPC-T6 board with a Rockchip RK3588, 16 GB of memory and a 6 TOPS NPU, the authors measure time to first token 4.41 percent above the same inference run without any protection, averaged over their prompt lengths. Against a TrustZone baseline that already uses a secure NPU and a loading pipeline, they report FlexServe 1.43 to 2.42 times faster, 1.85 times on average.

TrustZone splits the chip into a normal world, where Android and every app runs, and a secure world the normal world cannot read. FlexServe puts the inference engine, the weights, the prompt, the generated text and the KV cache on the secure side, so a kernel an attacker has taken over cannot read any of it out of memory, and the authors justify that assumption with Linux’s 40 million lines of code and 9756 CVEs. Both halves are worth stealing in their account, since a vendor’s fine-tuned weights cost real money to train yet sit on a stranger’s device, and the apps calling a shared model feed it chat history and screen contents. The authors name the boundary they cannot cross, which is that an app in the normal world still receives its own input and output there, so a compromised kernel can take those, and only the weights stay protected in that case.

Plain TrustZone is slow at this because it hands out secure memory as a few large contiguous regions. Reserving 8 GB at runtime therefore means the kernel’s contiguous allocator compacting scattered pages, which the authors time at 6440.67 ms against 568.58 ms for FlexServe’s page-granular secure memory, and the secure world cannot normally touch the NPU at all, leaving 30.06 s of CPU computation where the NPU needs 1.94 s. FlexServe carves secure pages out with the stage-2 page tables that ARM’s virtualisation extension already provides, and flips the NPU between protected and unprotected mode in 0.21 ms. Handing 8 GB back drops from 732.85 ms to 80.50 ms, because the pages are released lazily rather than zeroed in one pass.

A scheduler decides which models stay resident when several share the device, holding the earliest layers of each in secure memory so the pipeline does not stall and prefetching the next model while the current one is still generating. With a 4 GB cache and requests drawn at random from UltraChat, OpenAssistant, Dolly and Alpaca, the authors measure pairs of models answering 8.84 times faster than the plain secure-world baseline and 2.53 times faster than the optimised one. Across ten agent workflows, where one model’s finished answer feeds the next, the gains reach 24.30 times and 4.05 times, averaging 14.15 and 2.94. Serving one phone-side model to several callers is also what ElastiLM works on, by resizing the shared model to each request’s latency budget rather than isolating it.

The protection costs the rest of the device something as well. A SQLite instance holding 3 GB in the normal world runs 1.31 times slower while FlexServe holds 8 GB in the background, against 3.27 times slower with the contiguous allocator, and the authors put the standing virtualisation cost at 2.46 percent on SPEC CPU, which they switch off when no secure inference is running. They list the prototype’s limits themselves, a development board rather than a phone, Llama 3 and Qwen3 series models only, no GPU in the pipeline, and an NPU runtime slow enough that computation becomes the bottleneck, so a faster NPU would widen their lead. Side-channel attacks, physical attacks and denial of service are out of scope, and the paper names no code release.

Alibaba’s Qwen team released Qwen3.5-0.8B and Qwen3.5-2B on March 2, 2026, together with 4B and 9B versions, according to the Qwen release notes. Both are dense vision-language models with a 262,144-token context and Apache 2.0 weights. The model card states that the released files work with Hugging Face Transformers, vLLM, SGLang and KTransformers, and that in light of the parameter scale the intended uses are prototyping, task-specific fine-tuning and other research or development purposes.

The benchmark table in that model card puts Qwen3.5-2B at 55.3 on MMLU-Pro with reasoning switched off, where the same table gives 40.2 for last generation’s Qwen3-1.7B and 69.6 for the larger Qwen3-4B-2507, and Qwen3.5-0.8B at 29.7. With reasoning on, Alibaba reports 66.5 for the 2B and 42.3 for the 0.8B on the same test, and 51.6 against 11.9 on GPQA, a set of graduate-level science questions. The same table has Qwen3-1.7B ahead of both new sizes on IFEval, which checks whether a model obeys explicit formatting instructions, at 68.2 against 61.2 for the 2B and 52.1 for the 0.8B with reasoning off. Alibaba ran all of these figures itself.

Both sizes ship a vision encoder in the same checkpoint rather than as a separate release, and Alibaba’s vision table gives Qwen3.5-2B 64.2 on MMMU, a college-level image question set, above the 61.4 it lists for the previous Qwen3-VL-2B, with 49 for Qwen3.5-0.8B. The two models stack 24 layers in which three linear-attention blocks, whose memory cost does not grow with the sequence, alternate with one full-attention block. Input embedding and output layer are tied, so a single 248,320-entry matrix does both jobs, and Alibaba puts language coverage at 201 languages and dialects.

Alibaba published the weights on Hugging Face and ModelScope under Apache 2.0, and its MNN team put 4-bit exports of both sizes up the same day. The 0.8B build carries a 470 MB language weight file plus 63 MB for the vision encoder, and the 2B build 1.18 GB plus 196 MB, against 1.75 GB and 4.55 GB for the full-precision checkpoints. For other local runtimes the Qwen repository points readers to llama.cpp, which it says supports the series for text and vision, and to mlx-lm and mlx-vlm on Apple silicon, without Alibaba publishing GGUF or MLX builds of these sizes itself.

Seven researchers at MBZUAI, City University of Hong Kong and one working independently published ClawMobile on February 26, 2026, an agent runtime that runs on the Android phone itself with no tethering to a host machine. The model is not on the device. The authors state that ClawMobile runs locally while model inference is performed remotely, and all agents in their tests use GPT-5.2, so the split is an on-device runtime driving a cloud model. Across six real-life phone tasks the authors report ClawMobile completing all six at 100 percent, against 33 to 100 percent for the UI agent they compare against.

What separates it from screen-reading GUI agents is that the reasoning loop never touches the screen. The orchestrator issues tool calls to control backends instead, and the authors describe a deterministic-first policy in which the runtime first checks its memory for a structured interface, reaching for ADB system commands and the Termux hardware API before anything visual. A semantic UI agent is invoked only when a step genuinely depends on reading dynamic interface state, and raw taps on interface elements sit below that as a fallback. After every action the runtime re-queries device state to confirm the step landed, which the authors contrast with inferring execution success purely through model reasoning.

The tests ran on a Google Pixel 9 on Android 16, with DroidRun as the UI agent baseline and a stripped ClawMobile without DroidRun as a second baseline, all three on GPT-5.2 and scored by human annotators. In the authors’ table DroidRun reaches 100 percent on switching the system to dark theme and on playing a YouTube video, 85 percent on posting a YouTube comment, 73 percent on a Chrome search and on a cross-app task that searches football results and writes them into Notes, and 33 percent on installing an app from the Play Store, where ClawMobile scores 100 percent throughout. The authors put the cost of that at 57.5 seconds slower per task on average, with ClawMobile taking 21 seconds for the dark theme switch and 235 seconds for the YouTube comment. Their deterministic-only variant hits the 600-second timeout on both YouTube tasks.

The authors name the remote model as the main open problem, writing that fully remote inference introduces network latency and privacy concerns and calling on-device inference a promising direction they have not taken. They also flag that serialising full UI trees or screenshots into model context can dominate token use and latency on mobile, and present the results as preliminary, six tasks on one device. The code is open source on GitHub, built on the OpenClaw agent framework with a Telegram bot as the chat channel, and the paper was accepted at EuroMLSys 2026 under a Creative Commons Attribution 4.0 license.

Apple published python-apple-fm-sdk, Python bindings for its Foundation Models framework, with a first beta release on February 25, 2026 according to the repository’s release history. The package reaches the on-device model at the core of Apple Intelligence from Python on macOS, and installs with pip install apple-fm-sdk.

The Python API follows the Swift one. SystemLanguageModel reports whether the model is available and, if not, why, while LanguageModelSession holds a conversation and its async respond call returns text or streams it. A @fm.generable decorator marks a Python class for the model to fill in, and fm.guide adds per-field constraints such as a numeric range, as the README shows with a Cat class whose age is bounded to 0 through 20. Tool calling and session transcripts are exposed as well.

The bindings do not reimplement inference. The repository carries a Swift package named foundation-models-c that wraps the framework behind a C interface and builds it as a dynamic library, which the Python layer then calls. Apple writes in the documentation that the bindings run the Swift framework underneath, so evaluations reflect real on-device performance and behaviour, and its evaluation guide notes that inference calls are processed one at a time rather than in parallel at the macOS hardware level.

Apple lists macOS 26.0 or later, Xcode 26.0 or later with its agreement accepted in the Xcode app, Python 3.10 or later, and Apple Intelligence switched on. PyPI carries only source distributions for the package, so the Swift component is compiled locally during installation, which is what the Xcode requirement covers. Apple Intelligence on the Mac runs on Apple silicon machines only.

The code is under the Apache 2.0 license and the package metadata on PyPI classifies it as alpha. Apple pushed 0.1.0 to PyPI on March 8, 2026, added the Attachment API for sending images alongside text in 0.2.0 on June 8, 2026 for WWDC 2026, and exposed the model’s context size and the token count of a given input in 0.2.1 on June 29, 2026. The README states the project is not yet taking contributions.

Alibaba’s Tongyi Lab published Mobile-Agent-v3.5 on February 15, 2026, describing GUI-Owl-1.5, a family of agent models that read a screenshot and emit the next tap, type or scroll. The family spans 2B, 4B, 8B, 32B and 235B-A22B parameters in instruct and reasoning variants, all built on Qwen3-VL. Every benchmark in the paper was run server-side against cloud sandboxes and emulators, and the authors give no on-device deployment, latency or hardware figures anywhere, so the small sizes are the only thing that points at a phone. The authors write that the smaller instruct variants, which skip the reasoning step, are fast enough to be deployed on edge devices and to work alongside a larger cloud model.

Across platforms, the authors report their 32B-Instruct model reaching 56.5 on OSWorld-Verified for desktop tasks, 47.6 on OSWorld-MCP and 46.8 on Mobile-World, the last two of which also require calling external tools. On AndroidWorld, which scores an agent driving real Android apps in an emulator, their 8B reasoning model reaches 71.6, which the paper puts just behind UI-TARS-2 at 73.3. For browsers, the paper’s table has the 32B reasoning model at 48.4 on WebArena, 46.6 on VisualWebArena and 82.1 on WebVoyager.

The small checkpoints carry a large part of that. The authors report 2B-Instruct at 43.5 on OSWorld-Verified and 67.9 on AndroidWorld, and 4B-Instruct at 48.2 and 69.8, which puts the 4B model within 2 points of their own 32B on Android tasks. On ScreenSpot-Pro, which tests whether a model can point at the right element in a high-resolution screenshot, they report 72.9 for 32B-Instruct and 80.3 when the model first crops a region and looks again. The paper’s prose and its tables disagree on which variant owns several headline numbers, and the figures above follow the tables.

Six checkpoints are on Hugging Face under the MIT license, covering 2B, 4B and 8B instruct, 8B reasoning, and 32B in both variants. The 235B-A22B model named in the paper is not among them. The code is MIT as well, and the repository points at a hosted cloud-sandbox demo rather than a local runtime.

Ali Chherawalla has posted Off Grid to Show HN, an Android and iOS app that runs text generation, image generation, vision and speech recognition entirely on the device. The submission reached 124 points and 66 comments. The repository is licensed under MIT.

The app bundles llama.cpp for text, Stable Diffusion for images and whisper.cpp for speech to text. It accepts any GGUF file and lists Qwen 3, Llama 3.2, Gemma 3 and Phi-4 among supported models; vision runs on SmolVLM, Qwen3-VL or Gemma 3n. It also connects to OpenAI-compatible servers on a local network, and builds a project knowledge base from PDFs chunked and embedded on-device with a bundled MiniLM model into SQLite.

The README reports 15 to 30 tokens per second on flagship CPUs and 20 to 40 tokens per second through Adreno GPUs via OpenCL on a Snapdragon 8 Gen 2 or newer. Image generation is given as 5 to 10 seconds on the Hexagon NPU and vision inference at about 7 seconds. The NPU path is marked experimental: it accelerates only Q4_0 and Q8_0 quantisation, and K-quants fall back to the CPU.

Comments disputed whether models of that size are usable. One reader wrote:

Ive tried a lot of self hosted models at a lot of sizes; those small models are not good enough, and do not have a context long enough to be useful for most everyday operations.

Chherawalla replied that “you don’t need claude level performance for a lot of day to day tasks” and that Qwen3 8B, or 4B quantised, is “actually quite good”.

Other threads compared the app with PocketPal AI, SmolChat and Google’s AI Edge Gallery, asked for an F-Droid build, and reported that image generation required network access on GrapheneOS. A reader on a Samsung device reported the interface did not move for the on-screen keyboard; the developer shipped a fix the same day.

Samsung Research India has published NanoSD, a family of distilled Stable Diffusion 1.5 models for image restoration on edge devices. The variants range from 130M to 315M parameters, compared with 829M for the SD 1.5 baseline. The team built them with network surgery and feature-wise generative distillation, and they handle super-resolution, deblurring, face restoration and monocular depth estimation.

The paper measures latency on the Qualcomm SM8750 NPU in a Galaxy S25 Ultra, with 8-bit weights and 16-bit activations, and on the Apple A17 Pro Neural Engine. The 315M-parameter Prime model needs 27 ms per 128x128 tile, while SD 1.5 with INT4 quantization takes 116 ms. A 4000x3000 image splits into 88 tiles and finishes in about 1.8 seconds end to end.

On quality, the Nano-S3Diff variant reaches the best NIQE and MUSIQ scores for super-resolution among the compared methods, and Nano-OSDFace matches OSDFace on face restoration metrics. The authors state that parameter reduction alone does not correlate with hardware efficiency. The paper is published under a CC BY 4.0 license.

Nvidia published Nemotron-Flash on November 24, 2025, a family of small language models built around measured decoding latency instead of parameter count. Nvidia reports Nemotron-Flash-1B decoding 8k tokens in 14.45 seconds on a single Nvidia H100 at batch size 1, against 27.55 seconds for Qwen3-0.6B, while averaging 49.63 across 16 benchmarks to Qwen3-0.6B’s 44.11. The paper was accepted at NeurIPS 2025.

The models mix three kinds of layer rather than using attention throughout. Nvidia interleaves DeltaNet and Mamba2 layers, operators that carry a fixed-size state instead of a cache that grows with the context, with a small number of ordinary attention layers, and an evolutionary search picked which operator goes where. Nemotron-Flash-1B holds 0.96B parameters in 12 blocks with only 2 full attention layers, and Nemotron-Flash-3B holds 2.7B parameters in 18 blocks with 3. Nvidia built both wider and shallower than other models of the same size, where Qwen3-0.6B stacks 28 layers, and writes that deep and thin models are parameter efficient but sit off the accuracy-latency frontier.

Nemotron-Flash-3B averages 60.98 across the same 16 tasks, which cover MMLU, commonsense reasoning, math, coding and recall, against 58.96 for Qwen2.5-3B and 55.47 for Qwen3-1.7B. On the same H100, with all models running through TensorRT-LLM AutoDeploy kernels and CUDA Graph, Nvidia measured 28.71 seconds for the 3B against 49.40 for Qwen2.5-3B and 36.20 for Qwen3-1.7B, so 1.7 and 1.3 times lower latency. Throughput was measured separately, with a 32k-token input at the largest batch size that still fits in memory, and Nvidia reports 2939 tok/s for the 3B against 459 and 157 for the two Qwen models, and 7289 tok/s for the 1B against 160 for Qwen3-0.6B. Every figure here is Nvidia’s own, taken on data centre GPUs, and the paper gives no phone or embedded measurements, with the earlier depth and operator studies run on an Nvidia A100.

Nvidia trained both models on 4.5T tokens using 256 H100 GPUs and released three checkpoints on Hugging Face, Nemotron-Flash-1B, the 3B base model and an instruction-tuned Nemotron-Flash-3B-Instruct, which averages 52.84 across MMLU, GPQA, GSM8K and IFEval against 48.17 for Qwen2.5-1.5B. The model cards set the licence to Creative Commons Attribution-NonCommercial 4.0, so the weights are downloadable but not licensed for commercial use, and neither Apache nor an Nvidia open model licence applies. Running them needs the TensorRT-LLM AutoDeploy attention kernel alongside the Mamba2 and FlashLinearAttention implementations, since Nvidia states that vLLM and TensorRT-LLM do not support DeltaNet-style linear attention on their own.

Researchers at Stanford University and Together AI profiled more than 20 local models on 1M real queries and report that at least one model with 20B or fewer active parameters answered 88.7% of them correctly. The best single model in their set, GPT-OSS-120B, reached 71.4% on its own, averaged across the four query collections. Their 2025 efficiency figure comes from that model running on an Apple M4 Max.

The paper proposes intelligence per watt, mean task accuracy divided by the mean power the accelerator draws while answering, so a pairing that gets more answers right on the same power budget scores higher. The best local pairing each year moved from 7.92×10⁻⁴ accuracy per watt for Mixtral 8x7B on an NVIDIA Quadro RTX 6000 in 2023, to 1.80×10⁻³ for Llama 3.1 8B on an RTX 6000 Ada in 2024, to 4.18×10⁻³ for GPT-OSS-120B on an Apple M4 Max in 2025, which the authors put at a 5.3 times gain over two years. Query coverage for that single best pairing rose from 23.2% to 71.3% across the same three years. A companion metric divides accuracy by the energy a query consumes instead, so slow generation counts against a device.

The 1M queries come from four collections. The team took 500K prompts from WildChat, a month of real ChatGPT traffic, 500K from NaturalReasoning, 12K from MMLU Pro and 26.5K from SuperGPQA, all single-turn. WildChat has no ground truth, so GPT-4o judged whether a local model’s answer matched one written by Qwen3-235B, while the other three collections supplied reference answers. Every run used a batch size of 1, one query at a time.

The hardware the authors call local is desktop and workstation class rather than mobile. Their 2025 numbers ran on an Apple Mac Studio with an M4 Max, 128 GB of unified memory and 480 W of continuous power, and the 2023 and 2024 points used NVIDIA workstation cards. No phone, tablet or wearable was profiled. Running the same Qwen3 models, the M4 Max reached 1.40 times lower accuracy per watt than an NVIDIA B200 and 1.6 to 2.3 times lower accuracy per joule, a gap the authors read as headroom for local accelerator design.

For the savings case the team simulated 80.2M queries over 24 hours split between four small models on M4 Max machines and Qwen3-235B in the cloud. Perfect routing to the smallest capable model cut energy by 80.4% against sending everything to the frontier model, and a router that picks correctly 80% of the time cut it by 64.3%, with the perfect case described as a theoretical maximum. The authors note that coverage falls in technical domains such as architecture and engineering, that GPT-OSS-120B answered 88.9% of WildChat queries against 64.9% of NaturalReasoning ones, and that they release their profiling harness so the measurements can be repeated on new hardware.