All updates

Apple opened its on-device foundation model to third-party apps at WWDC25 on June 9, 2025. The Foundation Models framework gives any app direct access to the roughly 3-billion-parameter model, which runs offline and costs developers nothing per call.

Developers annotate Swift data structures with the @Generable macro, and the framework applies constrained decoding, which Apple calls guided generation, so the model returns those structures rather than free text. Tool calling lets the model invoke functions the app supplies. Developers who need more control can train rank 32 adapters with a Python toolkit.

Apple compressed the model to 2 bits per weight using quantisation-aware training. It splits the transformer into two blocks at a 5:3 depth ratio and shares the key-value cache between them, which Apple says cuts KV cache memory use by 37.5 percent. Images are handled by a 300M-parameter ViTDet-L vision encoder.

The model is designed to support 15 languages. The companion server model on Private Cloud Compute uses what Apple calls a Parallel-Track Mixture-of-Experts design, where independent transformer tracks cut how often the tracks have to synchronise with each other.

Vijay Janapa Reddi of Harvard University published an article in ACM Queue on May 21, 2025, arguing that generative models reach phones, glasses and robots only if the field stops chasing parameter counts and starts designing for deployment. He projects more than 50 billion edge devices by 2030 and writes that today’s frontier models, often hundreds of billions of parameters, cannot leave the data centre. His deployment matrix places only models under 1B parameters in the viable column for true edge hardware, puts medium models of 1B to 10B parameters within reach of high-end phones and single GPUs, and confines anything larger to servers or the cloud.

Reddi grounds that in memory first. He writes that a 6B parameter model at half precision needs roughly 12 GB for weights alone, far above typical mobile RAM, that even a 1B model at about 2 GB can strain some devices, and that small models usually occupy 100 MB to 2 GB. He puts the latency bar at around 50 ms for an augmented reality translation to feel instantaneous and under a second for a dialogue assistant to feel fluid, and he notes that batching across users, which keeps cloud serving efficient, does not apply when requests arrive one at a time. On energy he cites an int8 quantised model using about five times less energy per query than an FP16 one on the same hardware, and says MLPerf Power results suggest model energy efficiency is plateauing even as hardware improves.

Reddi frames the whole problem as a data-model-compute triangle, where scarce local data, limited compute and the need for compact models each worsen the others, and he argues compression alone will not close the gap, since smaller models hallucinate more and can lose their refusal behaviour when quantised hard. He proposes measuring capability and cost together through metrics such as hallucinations per watt-hour, running a model on factual question answering until it consumes 1 Wh and counting the wrong statements, and toxicity per token normalised by model size. The article ran in Queue volume 23, issue 2, and the full text is free to read on queue.acm.org.

Google previewed Gemma 3n on May 20, 2025, a model built for phones, tablets and laptops. Its 5B and 8B versions run with dynamic memory footprints of 2 GB and 3 GB, which Google says matches what 2B and 4B models normally need.

The saving comes from Per-Layer Embeddings, a Google DeepMind technique that keeps embedding parameters off the accelerator. For E2B that cuts what has to be loaded into accelerator memory from 5.44B parameters to 1.91B, with the 2.55B embedding parameters cached to fast storage instead. A MatFormer setup nests a smaller but fully functional 2B model inside the 4B one, so developers can trade quality against footprint. Google reports the model starting to respond about 1.5 times faster on mobile than Gemma 3 4B.

Gemma 3n takes audio, text, images and video as input, including interleaved inputs across modalities, and handles speech recognition and translation. Google states that the same architecture powers the next generation of Gemini Nano, which was to reach Android and Chrome later in 2025. Qualcomm Technologies, MediaTek and Samsung System LSI worked on the optimisation.

Update, June 26, 2025. Google released Gemma 3n as E2B and E4B, naming them for 2B and 4B effective parameters out of 5B and 8B raw ones. Google reports E4B scoring 1303 on LMArena and calls it the first model under 10 billion parameters to pass 1300, ahead of Llama 4 Maverick 17B 128E at 1292 and GPT 4.1-nano at 1288. The release adds a MobileNet-V5-300M vision encoder, which Google says handles up to 60 frames per second on a Pixel, and KV cache sharing that doubles prefill performance. Text covers 140 languages, multimodal understanding 35. Weights are on Hugging Face and Kaggle, with support in Ollama, llama.cpp, MLX, LiteRT and the Google AI Edge Gallery.

Google’s AI Edge team has published Google AI Edge Gallery, an app that runs generative models entirely on the device. The home screen offers three tasks, AI Chat for multi-turn conversation, Ask Image for questions about a photo from the camera or the gallery, and Prompt Lab for single-turn work, which covers free-form prompts, rewriting text in a chosen tone, summarising and generating code. Once a model file is on the phone, the app needs no connection and Google states that all processing happens on the device.

Models come from the LiteRT Community organisation on Hugging Face, which the app’s welcome text links to directly. The first release listed three downloads, Gemma 3 1B IT at 4-bit in 554.7 MB, Hammer 2.1 1.5B at 8-bit and Qwen2.5 1.5B Instruct at 8-bit, the latter two at roughly 1.6 GB each. Gated repositories ask for a Hugging Face access token, which the settings dialog stores and can clear again.

Anything outside that list has to arrive as a LiteRT .task bundle picked from device storage, and the app tells the user that no other file type is supported. Each model then carries a settings dialog with an accelerator switch between GPU and CPU, next to max tokens, top-k, top-p and temperature. The shipped list sets a different default per model, GPU first for Gemma 3 1B and CPU only for Qwen2.5 1.5B Instruct.

After every reply the app prints four of its own measurements, time to first token in seconds, prefill speed and decode speed in tokens per second, and total latency in seconds. It derives them from the session itself, dividing the prompt’s token count by the time the first token took and counting decoded tokens against the clock that starts with that first token, so the numbers describe whatever hardware the app happens to be running on.

Google calls this an “experimental Alpha release” and ships it under the Apache 2.0 licence. There is no store listing at this point, so installing means taking the APK from the project’s releases page on a device running Android 8.0 or newer, with a project wiki covering the corporate-device case. The README lists Android as available now and iOS as coming soon. Underneath, the app builds on LiteRT and Google’s MediaPipe LLM inference API for Android.

Alibaba’s Qwen Team published the Qwen3 Technical Report on May 14, 2025, covering six dense models from 0.6B to 32B parameters and two mixture-of-experts models, where only part of the model is active per token. The two smallest dense sizes, Qwen3-0.6B and Qwen3-1.7B, carry the same mode switch as the rest of the family, so one set of weights either works through a problem step by step before answering or replies straight away. Alibaba reports Qwen3-1.7B scoring 93.4 on MATH-500, a 500-problem maths set, with reasoning on, against 73.0 with it off.

Earlier Qwen releases needed two separate models for that, a chat model and a dedicated reasoning model such as QwQ-32B. In Qwen3 the user picks the behaviour with a flag in the prompt or the system message, /think or /no_think, and the report states that reasoning is the default and that across a multi-turn conversation the model follows the last flag it saw. The same section describes a reasoning budget, where the model’s working is cut off once it passes a token limit the user set and it answers from what it has so far, which the authors say they never trained for and which emerged from merging the two modes.

The benchmark tables give both small sizes in both modes. For Qwen3-0.6B, Alibaba reports 55.6 on MMLU-Redux, a corrected version of the MMLU knowledge test, with reasoning on against 44.6 with it off, and 77.6 against 55.2 on MATH-500. For Qwen3-1.7B the same pair of tables gives 73.9 and 64.4 on MMLU-Redux, 48.3 against 13.4 on the AIME 2024 competition maths set, and 33.2 against 11.6 on LiveCodeBench v5. Against the reasoning baselines in the report, Qwen3-0.6B is ahead of DeepSeek-R1-Distill-Qwen-1.5B on MMLU-Redux, 55.6 to 45.4, and behind it on MATH-500, 77.6 to 83.9.

Alibaba built the small sizes by copying behaviour from the large ones rather than by running the full reinforcement learning pipeline on each. The report describes two stages of distillation from Qwen3-32B and Qwen3-235B-A22B, first on stored teacher outputs generated in both modes, then by matching the small model’s output probabilities against the teacher’s while the small model generates, and it credits the first stage with teaching the smaller models to switch modes at all. Pre-training ran on about 36 trillion tokens covering 119 languages and dialects, up from 29 in Qwen2.5.

Context length is where the two small sizes sit apart from the family. Table 1 of the report puts Qwen3-0.6B and Qwen3-1.7B at 32K tokens against 128K for Qwen3-8B and every larger model, and both tie the input embedding to the output layer so one weight matrix does both jobs. All eight models are published under the Apache 2.0 license on Hugging Face, ModelScope and Kaggle. The Qwen3 blog post of April 29, 2025 names Ollama, LM Studio, MLX, llama.cpp and KTransformers as the tools it recommends for running them locally.

Eight researchers at Google published ML Drift on May 1, 2025, a GPU inference framework for running large generative models on the graphics chip of a phone, a laptop or a Mac. On the Adreno 750 GPU of a Samsung S24, the authors measure 1370 tokens per second of prefill and 37.1 tokens per second of decode for Gemma2 2B, and 412 and 12.7 tokens per second for Llama 3.1 8B. They report a 5 times to 11 times prefill speedup over llama.cpp, MLC LLM and the Qualcomm AI Hub benchmark on Qualcomm GPUs, and the work is due at the CVPR 2025 Workshop on Efficient and On-Device Generation (EDGE).

The team benchmarked five mobile GPUs, the Adreno 830 in a Xiaomi 15 Pro with 16 GB of RAM, the Adreno 750 in a Samsung S24 with 8 GB, the Adreno 740 in a Samsung S23 Ultra with 8 GB, the Arm Immortalis-G720 in a vivo X100 Pro with 16 GB and the Arm Mali-G715 in a Pixel 9 with 12 GB, each at a fixed 1024 prefill and 256 generated tokens. Llama 3.1 8B with every weight at 8-bit ran out of memory on the S24, the S23 Ultra and the Pixel 9 and finished only on the two 16 GB phones, at 7.70 and 4.72 tokens per second of decode. A mixed scheme that holds attention weights at 8-bit and drops embedding and feed-forward weights to 4-bit ran the 8B model on all five devices and lifted decode by up to 1.9 times. ML Drift also treats the two phases of generation separately, since filling the prompt is limited by arithmetic while producing each further token is limited by memory bandwidth.

A mobile GPU constrains inference in ways the paper works through one at a time. Memory comes first, and the authors report that Stable Diffusion 1.4 would need 4.31 GB for its intermediate 16-bit activations, which handing the same buffers to operations whose lifetimes do not overlap cuts to 387 MB. Weight layout comes second, and rearranging weight tensors into slices of four channels that match the 4-element units a GPU computes with buys up to a 20% speedup in matrix multiplication, which the authors call the primary driver of the framework’s advantage. Shader code comes third, and ML Drift writes its kernels once at startup from hand-tuned templates, so the translation between a tensor’s logical indices and the buffer or texture that actually holds it costs nothing while tokens are being produced.

The same engine renders Stable Diffusion 1.4 at 512 by 512 pixels over 20 iterations in 10.96 seconds on the S23 Ultra, under 9 seconds on the S24 and in 3.4 seconds on a laptop with an Intel Lunar Lake Ultra 7 258V. On an NVIDIA RTX 4090 the OpenCL path runs 5% to 25% slower than llama.cpp on CUDA, which the authors put down to NVIDIA’s Tensor Cores being unreachable through OpenCL and WebGPU. The paper names sub-channel quantisation, sparsity and vendor extensions for matrix multiplication as the next steps, and points to no public release of the framework.

Researchers at the Hong Kong University of Science and Technology and Shenzhen University published D2MoE on April 17, 2025, a serving system for mixture-of-experts models on edge hardware, accepted at ACM MobiCom 2025. On a laptop with a 6 GB NVIDIA RTX 3060 and on a Jetson AGX Orin with 64 GB, the authors measure Mixtral 8x7B at 1.14 to 1.39 times the throughput of EdgeMoE and LLaMA-MoE-3.5B at 1.06 to 1.16 times, the second one while using 33 to 53 percent less memory. Against MoQE-DynaIO, which quantises every expert to one fixed bit-width and loads them on demand, they report 1.42 to 3.37 times the throughput.

A mixture of experts replaces the feedforward block in each layer with a set of small networks, and a router sends every token to only a few of them, so the work per token stays small while every expert still has to be held in memory or fetched from somewhere. The paper puts Mixtral 8x7B at over 90 GB in float16, with experts accounting for 89.9 percent of the parameters, against the 6 GB the RTX 3060 laptop offers. One answer is to keep the whole expert set small enough to stay resident, the way Ai2 sizes OLMoE and Meta AI sizes MobileMoE. D2MoE takes the other one, quantising experts hard and loading them from lower tiers on demand, which is also how Edge0 fits a large model into a small budget.

That trade turns a memory problem into a transfer problem. On the RTX 3060 the authors time a single LLaMA-MoE expert at 3.1 ms of computation against roughly 20 ms of data transfer, so the GPU sits idle waiting for weights, and holding several quantised copies of each expert to choose from would give the memory back. D2MoE therefore routes each token twice, once to an expert and once to a bit-width for that expert, and nests the quantised weights so that a wider one contains the narrower ones, which the authors name after matryoshka dolls. They note that 4-bit LLaMA-MoE experts quantised with llama.cpp take 3.81 GB, while storing separate 2-bit, 3-bit and 4-bit copies takes 9.62 GB.

A scheduler then orders expert loads hottest first so that transfers overlap computation inside a set memory budget, and the paper puts the added cost of the second router at 0.28 percent of computation, 0.53 percent of memory and 1.67 percent of latency for LLaMA-MoE-3.5B. On WikiText2 the configuration meant to match 8-bit experts scores 4.09 perplexity on Mixtral 8x7B, against 4.04 for the baseline that holds all 8-bit experts in memory and 4.38 for EdgeMoE. Setting a model up is offline work, at about 2 hours of router fine-tuning plus 10 minutes of quantisation for LLaMA-MoE-3.5B and over 4 hours plus 20 minutes for Mixtral 8x7B. The prototype is roughly 2,500 lines of Python and CUDA on top of PyTorch, written for NVIDIA Ampere and Ada Lovelace, and the paper names no code release and reports no measurements on phones.

Adrien Grondin released Locally AI on April 13, 2025, an iPhone and iPad app that downloads open-weight language models and runs them entirely on the device. The App Store listing gives that release date and lists the app as free with no in-app purchases. Once a model is on disk the app needs no account and no internet connection, and Grondin states that no data is collected and nothing is sent to a cloud service.

The site at launch named Meta Llama 3.2 and Llama 3.1, Google Gemma 2, Qwen 2 and DeepSeek R1 as the models on offer, and described them as language and vision models, so a picture can go into the prompt alongside text. A user picks a model, downloads it once and can edit the system prompt the model runs under.

Grondin builds the app on MLX, Apple’s machine learning framework for Apple silicon, and attributes the performance to unified memory, where the CPU and the GPU reach the same memory pool so arrays need not be copied between them. The app fetches open-weight model files itself rather than calling any model that ships with the operating system.

Neither the site nor the App Store entry names a minimum chip. The site says only that Locally AI covers recent iPhone and iPad models and is optimised for Apple silicon, and the listing asks for iOS 18.1 or iPadOS 18.1 or later. No generation speed, memory requirement or model file size is given anywhere in the developer’s own material, and the launch site was a bought template whose team, testimonial and FAQ sections still carried placeholder Latin text.

Update, April 8, 2026. Grondin added support for Apple’s on-device Foundation Models framework in September 2025, so the app can open a conversation with the model built into iOS 26 without downloading anything, which Simon Willison recorded on September 21, 2025. A Mac version followed and requires macOS 26 or later. LM Studio announced on April 8, 2026 that Locally AI had joined the company and that Grondin had moved with it, and the App Store entry now reads Locally AI by LM Studio with Element Labs Inc as the seller.

Hugging Face published HuggingSnap, an iPhone app that describes what the camera is pointed at, with the model running on the phone rather than in the cloud. The app’s runtime configuration loads SmolVLM2 at 500M parameters in Apple’s MLX format. Hugging Face had trailed the app in its SmolVLM2 announcement on February 20, 2025, writing that the 500M model lets users analyse and understand video content directly on their device with no cloud required.

The README describes the app as a way to point the camera and have text translated or summarised, identify plants and animals and learn more about surrounding objects. Inference runs through MLX Swift, Apple’s machine learning framework for Apple silicon, with a modified fork of Apple’s MLX example code supplying the vision-language support. The app pulls the weights from the Hugging Face Hub the first time it runs, then holds Metal memory to 82 percent of what iOS reports as available to the process.

Input comes from the live camera, from stills and clips captured in the app, and from photos and videos picked out of the library. A tap on the shutter takes a picture and a press and hold records video, after which the app offers a describe action for images and a summarise action for video, with questions typed or dictated. The app’s own interface states that photos and videos are processed entirely on the device and that no data is sent to the cloud.

The README states that installation needs an iPhone running iOS 18, and the Xcode project sets iOS 18.0 as the deployment target and iPhone as the only device family, with no Mac Catalyst or visionOS build. TechCrunch reported the app as a free App Store download on March 19, 2025, quoting the listing as saying it works offline and processes all data on the phone. The SmolVLM2 weights are Apache 2.0, while the app repository itself carries no licence file.

Researchers at Shanghai Jiao Tong University, Peking University and Microsoft Research Asia described ROMA on March 17, 2025, an accelerator design that holds a quantised language model in on-chip read-only memory. The paper reports results from Verilog synthesis against TSMC’s 7 nm standard cell library, not from a fabricated chip and not from an FPGA prototype, so every speed, area and power figure in it is a toolchain estimate rather than a measurement on hardware. On that basis the authors report peak decoding of 31,800 tok/s for a 4-bit 3B Llama model with an empty KV cache.

The design splits storage by how often the weights change. The frozen base model, quantised to 2-bit or 4-bit, sits in ROM, which cannot be rewritten once the chip has been manufactured, while the small trainable LoRA adapter and the KV cache sit in SRAM, which can. The authors argue that a base model is stable enough for that, with updates expected over years rather than months, and that adaptation to new tasks happens in the adapter instead. What ROM buys is density, and the paper puts a ROM bit cell at roughly a third of the area of an SRAM bit cell in TSMC’s 7 nm memory compiler, enough to fit the whole quantised model on chip without external memory.

The synthesised configuration carries 1.86 GB of ROM and 304 MB of SRAM at 500 MHz, with an area of 503.7 mm² and power of 33.1 W, the area and latency coming from Synopsys Design Compiler and the power from Synopsys PrimeTime PX. It holds either a 4-bit 3B or a 2-bit 8B Llama model entirely on chip, and for the 3B model with a rank-16 adapter the authors report a time to first token of 5.6 ms at 256 input tokens and 140.2 ms at 4K, with decoding staying above 10,000 tok/s once 4K tokens are cached. They also compare against measured hardware, an Intel i5-1135G7 running llama.cpp at 6.8 tok/s and an Nvidia RTX 4090 running TensorRT-LLM at 219 tok/s on the same 4-bit 3B model, against 20,078 tok/s for the ROMA estimate at a 256-token input.

Most of the paper’s engineering goes into shrinking the ROM. A structure the authors call B-ROM stores the constant bits in the chip’s wiring instead of in transistors, which cuts the transistor count to about a quarter and the ROM area by around 40 percent against a standard ROM, and a fused cell layout places that wiring-heavy structure in the same physical area as the transistor-heavy compute logic. The paper states no tapeout, no fabrication schedule and no cost figures, and it names the fixed base model as the price of the approach, since ROM content is set at manufacture. It is posted under the arXiv perpetual non-exclusive license.

Google released Gemma 3 on March 12, 2025 in four sizes, 1B, 4B, 12B and 27B, and describes them as designed to run fast directly on devices, from phones and laptops to workstations. The 1B is a new size for the line, which started at 2B with Gemma and stayed there through Gemma 2. The technical report puts the 1B at 2.0 GB of weights in bfloat16 and at 0.5 GB as a quantised int4 checkpoint, and the 4B at 8.0 GB and 2.6 GB.

Context length is not the same across the four. The report gives 128K tokens for the 4B, 12B and 27B and 32K for the 1B, with the three larger models pre-trained at 32K and stretched to 128K at the end of pre-training by rescaling their rotary position embeddings. To stop the key-value cache from swallowing memory at long context, Gemma 3 places five local attention layers between each global one and holds the local layers to a 1024-token window, where Gemma 2 alternated one for one. Google reports that in an ablation on a 2B model a global-attention-only configuration carries a 60 percent memory overhead from the cache at a 32K prefill, and that interleaving with a 1024-token window brings that under 15 percent.

Multimodality starts at 4B. The 4B, 12B and 27B share one 400M-parameter SigLIP vision encoder, frozen during training, which takes square images resized to 896 by 896 pixels and condenses each one into 256 tokens for the language model to read. An inference-time step Google calls Pan and Scan cuts non-square or high-resolution images into equal crops and resizes each to the encoder’s resolution, which the report measures as raising the 4B from 72.8 to 81.0 on DocVQA and from 44.1 to 57.0 on InfoVQA in a 4-shot evaluation of a pre-trained checkpoint. The 1B has no vision encoder and takes text only.

Google published quantisation-aware trained checkpoints next to the raw weights, made by fine-tuning each model for about 5,000 steps against the probabilities of the unquantised model, so the weights are trained to tolerate the lower precision rather than rounded down afterwards. The three formats are per-channel int4, per-block int4 and switched fp8, which Google picked for what open source inference engines such as llama.cpp already support. The report’s footprint table counts weights alone and weights plus a key-value cache for a 32,768-token sequence, and on the second measure the 1B needs 1.4 GB at per-channel int4 against 2.9 GB in bfloat16, while the 4B needs 7.3 GB against 12.7 GB.

Google reports its own zero-shot figures for the instruction-tuned models and puts Gemma 3 4B at 43.6 on MMLU-Pro and 75.6 on MATH, against 15.6 and 27.2 for Gemma 2 2B and 56.9 and 55.6 for Gemma 2 27B, with the 1B at 14.7 and 48.0. The 4B also scores 48.8 on the MMMU validation set, a benchmark the text-only sizes do not run, and the report calls the instruction-tuned 4B competitive with the instruction-tuned Gemma 2 27B. Google put every size on Hugging Face, Kaggle and Ollama as a pre-trained and an instruction-tuned checkpoint under the Gemma Terms of Use, and names LiteRT, Gemma.cpp, llama.cpp and Google AI Edge among the runtimes that take them. The same release added ShieldGemma 2, a 4B image safety classifier built on Gemma 3 that labels images for dangerous content, sexually explicit material and violence.

Flower Labs released Flower Intelligence on March 12, 2025 as a preview, an open source inference library that runs a language model on the user’s own device and only moves the work to a remote service if the app and the user allow it. The company shipped SDKs for Swift and TypeScript, and the TypeScript build defaults to Llama 3.2 1B in 16-bit precision.

The library picks a local engine from the environment it finds itself in. The TypeScript package depends on MLC’s WebLLM, which executes models in the browser through WebGPU, and on Hugging Face transformers.js, which it uses when the code runs under Node.js rather than in a page. The Swift package is built on MLX Swift and Hugging Face swift-transformers, and declares iOS 16 and macOS 14 as its minimum platforms, so local inference on Apple hardware goes through MLX.

Flower Labs’ model reference covers Llama 3.2 at 1B and 3B, Llama 3.1 8B, SmolLM2 at 135M, 360M and 1.7B, and DeepSeek R1 Distill Llama 8B, most of them as 4-bit builds. Which of those run on device depends on the backend. Node.js is limited to the 1B and SmolLM2 entries, while the 8B models run only in the browser and on the MLX Swift path, and Llama 3.2 3B in 16-bit is MLX Swift only.

Remote handoff is off by default. An application has to set the remoteHandoff property and supply an API key before the library will contact Flower Confidential Remote Compute, and even with both in place a chat call tries a local engine first and retries remotely only when no local engine matches the model or the local run fails. Individual requests can be pinned with forceLocal or forceRemote. Flower Labs describes the remote service as a private extension of the device that costs the developer no extra work, and named Mozilla Thunderbird as an early adopter building its Thunderbird Assist feature on the library. Ryan Sipes, Managing Director for Mozilla Thunderbird, tied that choice to the mail client’s user base in the announcement.

Our 20 million users expect data privacy from every feature we build.

Both SDKs are published under the Apache 2.0 license, the TypeScript one as @flwr/flwr on npm and the Swift one as a package in the Flower repository. Flower Labs called the release inference-only and listed fine-tuning, RAG and pre-training as later steps, with the remote compute service available on application for early access.