All updates

Maximilian Abstreiter, Sasu Tarkoma and Roberto Morabito of the University of Helsinki and EURECOM measured 11 language models from 0.5B to 9.2B parameters on a Raspberry Pi 5 and an NVIDIA Jetson Orin Nano Developer Kit, both with 8 GB of RAM, running every model through llama.cpp at 4-bit. On the Pi, the authors report that only models up to Llama 3.2 3B, or up to Phi 3.5 mini in the faster of the two 4-bit formats they tested, generated text above 5.3 tokens per second, the rate they derive from average English reading speed. The Orin’s GPU stayed above that rate for all 11 models, its CPU cores up to InternLM 2.5 7B in that same format.

Generation speed drops off steeply with model size in the paper’s charts. Qwen 2 0.5B runs at roughly 46 tokens per second on the Orin’s CPU cores and roughly 33 on the Pi, while Phi 3.5 mini at 3.8B parameters sits at about 5 on the Pi, about 10 on the Orin CPU and about 15 on the Orin GPU. The authors also contradict Meta’s claim that Llama 3.2 is optimised for Arm processors, since the advantage they saw for Llama 3.2 1B showed up in GPU inference and not on either CPU.

Memory is where the measurements turn hardest. Peak usage grows close to linearly with model size, reaching about 7.9 GB for Gemma 2 9B on the Pi, and it flattens out on the Orin above Phi 3.5 mini because under 7 GB of the nominal 8 GB is usable once the operating system and background processes take their more than 1 GB. Raising the context window to 1028 tokens pushed the largest model into constant page faults on the CPU and made the Orin crash and restart during GPU inference. First-run load times ran from 5 seconds to over 200 seconds, and for Yi 1.5 6B and up the faster 4-bit format raised load times by 17 to 53 times, because those models get evicted from memory and re-read from the microSD card for every inference.

Scaled to the 18.84 Wh battery of a Samsung Galaxy S24 Ultra at full utilisation, the March version of the paper puts Gemma 2 9B at 193 inference runs on the Pi and 101 on the Orin GPU, against 3,928 and 4,280 runs for Qwen 2 0.5B, with no configuration lasting 7 hours. The setup was built to keep thermal throttling out of the results rather than to observe it, so the authors fitted the Pi with an add-on heatsink after it overheated in early experiments, ran the fans on both boards at full speed and left 30 seconds between runs.

Quantisation cost measurable quality. Perplexity, which tracks how uncertain a model is about its next token, rose against the 16-bit weights for every model and both 4-bit formats and rose most sharply for the smallest models, while the downstream benchmarks showed almost no accuracy drop. The paper is on arXiv and under review at an ACM journal.

Update, November 20, 2025. The authors revised the paper after validating their power measurements against external hardware, which raised the measured consumption. The battery estimate for Gemma 2 9B falls from 193 runs to 75 on the Pi and from 101 to 73 on the Orin GPU, and no configuration now reaches 3.5 hours at full utilisation. The speed, memory and quality results are unchanged.

Researchers at Pengcheng Laboratory, the University of Houston, Stevens Institute of Technology, Keio University and the University of Waterloo posted MobiLLM on arXiv on February 27, 2025, a way to fine-tune a language model on a single device while a server does the training work. Their OPT-1.3B table puts the device-side memory at 4.50 GB, against 14.57 GB for LoRA, 13.37 GB for BitFit, 10.80 GB for the LST side-tuning baseline and 20.90 GB for full fine-tuning, all at a batch size of 16 and a sequence length of 256. The device is an NVIDIA Jetson Xavier NX whose GPU can use 4.6 GB of the board’s 8 GB of shared RAM, so MobiLLM is the only method in the authors’ comparison that fits a billion-parameter fine-tune onto it. For OPT-350M the same table gives 1.64 GB against 6.70 GB for LoRA and 7.91 GB for full fine-tuning, which is 0.01 GB more than running that model for inference on the device.

Side tuning puts the two halves of training in two places. The device keeps the pre-trained backbone frozen and runs only the forward pass through it, while the server holds a small trainable side network built from adapter modules, each one a projection down to a narrow dimension, a non-linear step and a projection back up. The adapters sit parallel to the backbone rather than inside it, so gradients never travel back through the frozen model, and the authors trace the saving to the device storing neither the per-layer activations that backpropagation would need nor any optimizer state. It is a different bargain from splitting one model across several nearby devices, as Confidant does, which the authors argue needs a stable set of peers and leaves no single device holding a model it can run on its own.

What crosses the network is one-way and happens only during the forward pass. The device quantises the intermediate activations of selected backbone blocks, by default to 4 bits, and sends them to the server along with the labels for the batch and metadata such as batch indices, and nothing travels back while training runs. Raw samples stay on the device, which is the authors’ privacy argument, though the labels do leave it, and the paper states that the activation outputs of each block distort the input embeddings enough to make the original samples hard to infer, without reporting a reconstruction attack that tests this. Quantising those activations cuts what has to be sent per iteration for OPT-1.3B from 400 MB to 100.2 MB, at a GLUE average of 81.3 in the NF4 format against 81.5 unquantised.

The authors put the speed gain at 1.8 times to a target test accuracy on the Xavier and 2.3 times on a CPU-only Huawei MateBook laptop with a Core i5-13500H, both measured against LoRA, over a Wi-Fi 5 link to a server with an NVIDIA A100. Their link-rate table for OPT-350M is the more direct measurement, putting one iteration at a batch size of 16 at 7.48 s over 60 Mbps against 8.4 s for training on the device alone, and at 9.8 s when the uplink drops to 10 Mbps, where they say transmission becomes the bottleneck. Accuracy is the cost they name, with MobiLLM averaging 81.3 across the eight GLUE tasks on OPT-1.3B against 83.6 for full fine-tuning and 83.0 for LoRA, and 78.8 against 80.7 on OPT-350M.

Fine-tuning this way still needs a reachable server, and the authors point at 5G and 6G base stations and home Wi-Fi hubs as where one would sit rather than testing either. They name two next steps, quantising the frozen backbone weights themselves, for which a side experiment puts device-side memory at 2.751 GB for OPT-1.3B at INT4 precision against a 2 percent accuracy drop and at 1.176 GB for OPT-350M, and filtering which activations get sent at all to cut the upload further. The paper is posted under the arXiv non-exclusive distribution license rather than a Creative Commons one, and it names no code release.

Seven researchers at Johns Hopkins University and Duke University published a survey of generative AI on edge devices on February 19, 2025. Most of the accelerator hardware it collects has not been built. Its table of 12 accelerators for language and diffusion models covers 8 simulator studies against 4 fabricated chips, and the highest peak energy efficiency in it is 74.34 TOPS/W, from a 28nm computing-in-memory chip designed for Stable Diffusion.

Compression is the part the survey treats as settled enough to quantify, and it attributes every figure to the work it reviews. It reports unstructured pruning reaching up to 60 percent sparsity on large models and a sparsity-aware inference system turning that into a 3 times speedup, while structured pruning that removes whole components gives about 2 times with little quality loss. Quantisation has been taken to 1 bit per weight in work such as BitNet, and the survey names what is still open there as holding generation quality under extreme compression and finding efficient ways to train quantised models on the edge devices themselves rather than only to run them.

The frameworks chapter is where the survey names hardware a reader can buy. It cites a measurement that applying NVIDIA’s TensorRT to transformer models on a Jetson Xavier cuts latency by over 60 percent with negligible accuracy loss, and points to MELTing Point, a benchmark suite that tracks language model energy use and memory footprint across smartphones and Jetson boards. The authors state that no dedicated survey of generative AI at the edge existed before theirs, close by naming model personalisation and security across distributed edge nodes as the challenges that persist, and publish on arXiv under a Creative Commons Attribution 4.0 license.

Meta published ParetoQ on February 4, 2025, a training framework that puts 1-bit, 1.58-bit, 2-bit, 3-bit and 4-bit quantisation through the same recipe so the bit widths can be compared against each other rather than against methods built for one width. The result the authors draw from it is that 1.58-bit, 2-bit and 3-bit land close together and generally beat 4-bit on accuracy against model size, while 1-bit drops away. On LLaMA-3 8B the paper’s table gives an average of 71.2 across five commonsense tasks at 2 bits and 69.0 at 1.58 bits against 74.6 for the full-precision model, then 64.1 at 1 bit.

That ranking only holds when size is counted in bytes rather than parameters, which is how a phone’s memory budget works. The authors measure each model by its effective quantised size, every weight counted at its own bit width plus the embeddings at theirs, and compare across that axis. Their example is a 2-bit MobileLLM-1B scoring 61.0 on an eight-task average against 59.2 for a 4-bit MobileLLM-600M, 1.8 points higher at a smaller effective size, and the paper reports the same pattern across LLaMA models up to 8B.

The paper also describes where the training behaviour changes. At 3 bits and above the fine-tuned weights stay within 10 to 20 percent of the full-precision weights they started from and recover most of the accuracy after about 10B tokens, while at 2 bits and below they move by roughly 40 percent and need around 30B tokens, which the authors read as the network rebuilding its representations instead of nudging them. On that basis they argue against training ternary models from scratch, the route taken by BitNet b1.58 and Spectra, and put their ternary 600M model at 58.7 on a six-task average, against 57.5 for the ternary 3B model of the 1-bit Era and 58.2 for Spectra’s ternary 3.9B, both computed from the same table.

For speed the authors wrote a 2-bit CPU kernel and timed five MobileLLM models on an Apple M1 MacBook Pro with 32 GB of memory using 6 threads, 5 prompt tokens and 122 generated tokens, and report the 2-bit curve sitting above the 4-bit one on accuracy against decoding speed without naming a factor. The one speedup they put a number on is on an H100 NVL GPU, where their 2-bit kernel built on CUTLASS runs 4.14 times faster than FP16 and 1.24 times faster than the 4-bit Machete kernel from vLLM at a 16384 by 16384 weight shape. At 1B, 3B and 8B the authors say the in-kernel conversion overhead cancels that gain and 4-bit gives the better speed-accuracy trade-off.

Hardware is also why the paper favours 2-bit over ternary despite the two sitting together on its accuracy-size front. Storing three values needs either sparsity above 90 percent to pay for the indexing or packing that complicates the matrix multiply, and the authors note that some implementations store ternary weights as 2-bit signed integers anyway, which gives up the storage saving. They write that broad adoption will need “community-wide efforts, such as INT2 support in NVIDIA tensor cores”, and the paper itself names no weight release.

Update, May 30, 2025. Meta published the ParetoQ checkpoints on Hugging Face, five MobileLLM sizes from 125M to 1.5B, each as a BF16 model plus 1-bit, 1.58-bit, 2-bit, 3-bit and 4-bit versions, 30 repositories in all. They sit behind a gated form under Meta’s FAIR Noncommercial Research License, which permits research use only. The training code had gone up on GitHub on March 3, 2025.

Alibaba published the first public build of its MNN-LLM Android app on January 23, 2025, a client that downloads open models and runs them entirely on the phone, covering text-to-text, image-to-text, audio-to-text and text-to-image generation through diffusion models. It sits on MNN, the inference and training framework Alibaba has kept open source since 2019 and uses in more than 30 of its own apps, among them Taobao, Youku and DingTalk. Alibaba reports the runtime prefilling 8.6 times faster than llama.cpp on an Android CPU, with decoding 2.3 times faster.

Models reach the device through llmexport, which converts a PyTorch checkpoint to ONNX and then to MNN’s own format, splitting the weights into a separate file and quantising them to 4-bit or 8-bit by default, with block size configurable and 5-bit and 6-bit available through the MNNConvert tool. The same README lists Qwen, Gemma, Llama including TinyLlama and MobileLLM, Baichuan, Yi, DeepSeek, InternLM, Phi, ReaderLM and SmolLM as supported families. LoRA adapters can be merged into the base model at export or kept as a separate file that the runtime loads and swaps while running.

The MNN-LLM paper, presented at the ACM Multimedia Asia 2024 workshops, describes a DRAM-Flash split that leaves the embedding table, roughly 15% of the weights, in bfloat16 on flash storage and reads only the single token needed for each decode step. The authors put the saving for Qwen2 7B at about 2.18 GB of DRAM. Layers and the LM head stay in memory at int4 or int8 with asymmetric quantisation, computed as W4A8 or W8A8 on CPU and W4A16 or W8A16 on GPU, while the KV cache quantises keys to int4 or int8 and values to fp8 before spilling to flash with prefetching. The team measured the speed comparisons on a Xiaomi 14 against llama.cpp, MLC-LLM and fastllm, using Qwen2 1.5B, Qwen2 7B and Llama 3 8B.

Android builds run on the CPU with ARMv8.2 fp16 instructions and add an OpenCL path for the GPU, and the same runtime compiles for iOS with Metal, for macOS, Linux and Windows, and for browsers through WebAssembly. Alibaba distributes the app as a downloadable APK and warns that the first version was tested only on a OnePlus 13 and a Xiaomi 14 Ultra, so low-spec phones may run slowly or not at all. MNN is published under the Apache 2.0 license.

Shreyas Subramanian, Vikram Elango and Mecit Gungor of Amazon surveyed about 160 papers on small language models on January 3, 2025, covering general-purpose models under 8B parameters plus a few exceptions up to 13B. Their central claim is that some of these models reach the performance of models 10 to 100 times their parameter count, a ratio the authors call effective size and anchor to PaLM at 540B parameters. In their table, Phi-1 at 1.3B parameters is credited with an effective size of 175B on code benchmarks and Phi-2 at 2.7B with 70B.

The survey is clear about where that stops. Its appendix prints completions from 14 models for the prompt “The diagonals of a rectangle”, where a 33M TinyStories model answers “and they were so beautiful” and a 125M GPT Neo model loops on its own words, while TinyLlama at 1.1B and Phi 1.5 at 1.3B give the correct definition. The authors conclude that several billion parameters are the minimum for usable general-purpose responses, and they caution that every effective size they report comes from each paper’s best reported benchmark, so results on other benchmarks can be much lower. The clearest wins they list are task-specific and stay inside one domain, with WizardMath at 7B beating Llama 2 70B on maths benchmarks and Code Llama at 7B beating it on coding benchmarks.

On what closes the gap, the survey puts training data first. It credits Phi-1 with 50.6 percent on HumanEval at 1.3B parameters trained on textbook-quality synthetic data, and TinyGSM, where a 1.3B generator paired with a 1.3B verifier scored 81.5 on the GSM8K maths set against 77.4 for the GPT-3.5 model that wrote its training data. Distillation covers the training side, with Orca 2 at 13B leading Llama 2 70B by an average of 3.23 points after learning step-by-step explanations from a teacher model rather than just its answers. After training, the survey groups quantisation and pruning, citing SmoothQuant for 8-bit weights and activations at up to 1.56 times the speed and half the memory, and Sheared-LLaMA for cutting Llama 2 7B down to 1.3B and 2.7B using 3 percent of the compute that training those sizes from scratch would need.

The authors name resource-constrained deployment as the main motivation for the whole field, listing mobile devices, edge computing and regions with limited connectivity, and point at SmolLM at 135M, 360M and 1.7B parameters as a family built to run locally on phones and laptops. They note that extreme quantisation and sparse training already put Llama inference on CPUs within reach, and that 8-bit Adam training of a 7B model fits on a single 24 GB Nvidia RTX 4090. The paper is on arXiv under a CC BY-NC-ND 4.0 license, and its authors maintain a leaderboard of small models on Hugging Face Spaces.

EXO Labs opened a public benchmark site on December 25, 2024, as the first entry in a twelve-part series, publishing generation speed and time to first token measured on real consumer devices. EXO states these are measurements rather than estimates. In the data behind the site, an iPhone 15 Pro with 8 GB of RAM generates 14.0 tok/s on Llama 3.1 8B and takes 1.67 seconds to return its first token.

A Galaxy S24 Ultra with 12 GB manages 5.6 tok/s on the same 8B model, with 4.16 seconds to the first token, according to EXO’s published results. On the smaller Llama 3.2 1B the iPhone reaches 56.1 tok/s and the Galaxy 22.5 tok/s. A single Mac mini M4 Pro with 24 GB sits at 49.3 tok/s on Llama 3.1 8B, which puts the phone at under a third of the Mac’s rate on the same model.

EXO splits a model into contiguous slices of its layers and hands each slice to a different device, so a model too large for one machine can run across several. The activations passed from one device to the next are small, under 4 KB for Llama 3.2 3B, and EXO says the bottleneck is network latency rather than bandwidth. Pooling phones does not speed up a single reply. A cluster of two iPhone 15 Pros and one Galaxy S24 Ultra runs Llama 3.1 8B at 7.0 tok/s, below the 14.0 tok/s of one iPhone alone, which EXO attributes to network overhead when the model already fits on a single device.

Three Mac mini M4 Pros run Llama 3.3 70B at 4.1 tok/s, a model that does not fit in the 24 GB of any one of them. Every entry carries a timestamp, a git commit and the identifier of the automated run that produced it, and the site is free to browse. EXO says it is expanding the suite towards mixed device types and different quantisations, and the exo runtime itself is on GitHub under Apache 2.0.

Researchers at CUHK MMLab and vivo AI Lab published BlueLM-V-3B on November 16, 2024, a vision-language model designed together with the code that runs it on a phone. The authors deployed it on the NPU of a MediaTek Dimensity 9300 in a vivo X100 and report a peak memory requirement of 2.2 GB and a generation speed of 24.4 tok/s with the language model weights quantised to 4-bit.

The model pairs a 2.7B in-house BlueLM language model with a 400M SigLIP vision encoder that takes 384 by 384 image tiles. The authors report an average score of 66.1 on OpenCompass, a suite of eight multimodal benchmarks, which they state is the highest among models of 4B parameters or fewer, ahead of the 8B models MiniCPM-V-2.6 and InternVL2-8B.

Most of the speed comes from how the model decides what resolution to feed the encoder. Dynamic resolution schemes pick a grid of tiles to fit an image, and the authors show that existing choices enlarge pictures a lot, with LLaVA-NeXT resizing and padding a 394 by 390 image to 768 by 768, four times the area, and InternVL 1.5 stretching a 380 by 76 image to 1920 by 384, twenty-five times the area. More pixels mean more tiles and more image tokens, so BlueLM-V-3B uses what the authors call relaxed aspect ratio matching, which tolerates a slightly smaller effective resolution when it wastes less of the canvas. Both examples then land on a single 1:1 tile, and across the 665k-sample LLaVA training set the method picks a smaller grid than InternVL 1.5 in 523k cases.

The rest is shaped around what an NPU is bad at. The authors encode image tiles in batches of four at a time, which they measured as the fastest setting, and run the SigLIP convolution layer on the CPU in parallel with the transformer blocks on the NPU to hide 200 ms of latency. A downsampler merges every 2 by 2 patch into one token, cutting 729 tokens per tile to 196, and the prefill runs in chunks of 128 tokens rather than all at once. Precision is mixed, with INT4 for the language model weights, INT8 for the vision encoder and projector weights and for the KV cache, and INT16 and FP16 activations, which the authors say keeps accuracy stable because activations are more sensitive to quantisation than weights.

Against the 8B MiniCPM-V 2.5 running on the CPU of the same chip through llama.cpp, the authors report 2.53 s to load and encode a 768 by 1536 image and 2.7 s of prefill, versus 4.0 s and 13.9 s, at 24.4 tok/s against 4.9 tok/s. The paper describes no public release of weights or code, and the authors say future work will target a broader range of mobile devices.

Researchers at Beijing University of Posts and Telecommunications published PhoneLM on November 7, 2024, a small language model family in 0.5B and 1.5B sizes built in an unusual order. The team first searched transformer configurations for the fastest inference on a specific phone, then pre-trained the configuration that won, instead of choosing a shape for capability and optimising speed after the fact. PhoneLM-1.5B decodes at 58 tok/s on the CPU of a Xiaomi 14 with a Qualcomm Snapdragon 8 Gen 3.

The search varied the layer count from 15 to 25, the ratio of intermediate to hidden size between 2 and 5 and the attention type, and the paper lists measured throughput for seven candidate 1.5B shapes on that phone. The winner has 19 layers and a hidden size of 2,560, a hidden size to layer ratio of 134.7 against the 54.6 to 88.6 the authors observe in recent small models. It uses ReLU rather than the SiLU activation most current small models adopt, which the authors argue is cheaper on NPUs built for integer arithmetic and leaves more zeros in the feed-forward output for sparse-computation tricks. The paper notes that only the 1.5B model went through the search.

Across seven zero-shot tasks including HellaSwag, PIQA, BoolQ and both ARC sets, the benchmark table puts PhoneLM-1.5B at an average of 67.31%, ahead of SmolLM-1.7B at 66.49% and StableLM 2 1.6B at 66.34% and behind DCLM-1.4B at 68.33% and Gemma 2 2B at 71.31%. PhoneLM-0.5B averages 60.43% against 60.17% for SmolLM-360M and 56.46% for OpenELM-450M. The authors measure PhoneLM-1.5B decoding 1.2 times faster than StableLM 2 1.6B and 1.6 times faster than SmolLM 1.7B on the same handset. The two models saw 1.1T and 1.5T training tokens, all of them from open datasets such as DCLM-baseline, StarCoderData and Dolma.

The group ran every device measurement through its own inference engine mllm, with the linear and embedding weights quantised to 4-bit and four threads pinned to the phone’s performance cores. On the NPU it goes through Qualcomm’s QNN framework, where the authors report prompt processing for PhoneLM-1.5B at 654 tok/s on the Xiaomi 14 against 602 tok/s for Qwen 2.5 1.5B. The 4-bit mllm build the group publishes is a 1.41 GB file. The paper states no peak memory figure.

An Android demo app on top of mllm runs PhoneLM-1.5B locally for chat and for Android intent invocation, so a user who asks to be woken at 8 o’clock has an alarm set through the system’s own alarm intent. That variant, PhoneLM-1.5B-Call, was fine-tuned with LoRA on DroidCall, 10,000 synthetic function-calling examples covering alarms, timers, email drafts and searches, and the paper places it above GPT-4o-mini, which scores 71.0% on the same test set. Weights are on Hugging Face under Apache 2.0 and the data preparation, training, fine-tuning and evaluation code is on GitHub under GPL-2.0, alongside the list of open datasets used at each stage.

Hugging Face released SmolLM2 on October 31, 2024, a second generation of its small language models at 135M, 360M and 1.7B parameters. The company trained the 1.7B model on 11 trillion tokens, against 1T for the first SmolLM at the same size, and its model cards put the 360M model at 4T tokens and the 135M at 2T.

The mixture combines FineWeb-Edu, DCLM and The Stack with datasets Hugging Face curated for this run, which its paper from February 2025 names as FineMath for mathematics, Stack-Edu for code and SmolTalk for instruction tuning. That paper states the 1.7B run used a multi-stage schedule that held the highest quality maths and code data back for the final tenth of training, from 10T to 11T tokens, while the 360M and 135M models trained in a single stage on a mix kept consistent throughout. Hugging Face fine-tuned the instruct versions on SmolTalk and then tuned them further on ranked answer pairs from UltraFeedback.

Hugging Face reports the 1.7B base model at 60.5 on ARC against 49.2 for Llama 3.2 1B and 58.5 for Qwen2.5 1.5B, 36.7 on TriviaQA against 28.1 and 20.9, and 19.4 on MMLU-Pro against 11.7 and 13.7, all run zero-shot with its own lighteval harness and printed in the model card. The instruct version scores 56.7 on the IFEval instruction-following test against 53.5 for Llama 3.2 1B Instruct and 47.4 for Qwen2.5 1.5B Instruct, while Qwen2.5 stays ahead on the MT-Bench chat test at 6.52 to 6.13. At the smaller sizes Hugging Face compares the 360M model with Qwen2.5 0.5B, reporting 53.0 against 45.4 on ARC and 16.9 against 4.3 on TriviaQA, and publishes no third-party comparison for the 135M model.

The weight files run 269 MB for the 135M model, 724 MB for the 360M and 3.42 GB for the 1.7B in bfloat16, and Hugging Face uploaded 4-bit ONNX builds the same day at 118 MB, 273 MB and 1.11 GB, plus a 4-bit GGUF of the 1.7B instruct model at 1.06 GB. The repository says the models run locally through llama.cpp, MLX, MLC and transformers.js. All six checkpoints carry an 8k context window and the Apache 2.0 license.

Update, November 27, 2024. Hugging Face published a browser chat demo that runs the 1.7B instruct model through transformers.js and ONNX Runtime Web on WebGPU, with no server call after the download. Its local inference guide adds runnable code for llama.cpp, MLX and MLC, and lists the models on Ollama and in the PocketPal-AI phone app.

Seven Apple researchers measured what it costs to train language models up to 2B parameters on rented Nvidia GPUs and found the most expensive card was the worst buy. The H100-80GB returned the fewest tokens per dollar of the three cards tested at every one of their four model sizes, 100M, 500M, 1B and 2B, while costing 2.53 times as much per hour as the A100-40GB.

The price scale comes from normalising published hourly rates at Lambda Labs and Google Cloud, with the A100-40GB set at 1.00 and the A100-80GB landing at 1.46. The same comparison shows cost efficiency falling as GPUs are added, with a single card the best or near-best point at all four model sizes. For the 1B model, tokens per dollar drop from about 17,000 on one GPU to about 10,300 on 64.

FlashAttention mattered more at small scale than at large. Reading the authors’ first figure, the 100M model gained roughly three times the tokens per dollar over vanilla attention, while at 2B the gain was under two times. The authors put this down to the cost of attention being quadratic in context length, so it dominates once the hidden dimension shrinks and the small models become bound by data movement between CPU and GPU and between GPUs. Vanilla attention also ran out of memory at a global batch size of 1024 on the 1B and 2B models, where FlashAttention did not.

Two habits carried over from large-model training did not pay off. Plain Distributed Data Parallel, which keeps a full copy of the model on every GPU, beat both sharded FSDP variants for the smaller models because it moves less data, and FSDP only pulled ahead at 2B, where it kept training at a global batch size that made DDP run out of memory. Filling the cards was not cost-optimal either, since the batch size figure peaks around a global batch size of 128 for the 100M model, 32 for 500M, 16 for 1B and 8 for 2B, in every case well short of what the memory allowed.

These are training runs on datacentre GPUs, not anything running on a phone, and the authors frame the work as guidance for labs with small budgets rather than for deployment. They name their own limits, running each setup at least three times for only 10 training steps, reporting normalised price ratios rather than absolute dollars, using no gradient accumulation, and leaving GPU utilisation and memory bandwidth measurements to future work. They also assume that settings such as the learning rate can be retuned to converge at whichever hardware configuration turns out cheapest. The 8-page paper is free to read on arXiv under the site’s non-exclusive distribution licence.

Mistral AI introduced Ministral 3B and Ministral 8B on October 16, 2024, two models the company built for on-device computing and at-the-edge use and announced together as les Ministraux. Mistral says Ministral 3B already outperforms its own Mistral 7B on most benchmarks, and reports the base model scoring 60.9 on MMLU against 52.4 for Gemma 2 2B and 56.2 for Llama 3.2 3B.

Both models handle up to 128k tokens of context, which the announcement says was capped at 32k when run on vLLM. Ministral 8B uses what Mistral calls an interleaved sliding-window attention pattern, where a layer attends only to a limited span of recent tokens instead of the whole sequence, which the company says makes inference faster and lighter on memory.

Mistral reports the instruction-tuned Ministral 3B at 8.1 on MTBench, 64.3 on Arena Hard and 77.4 on HumanEval, the last of those above Gemma 2 2B at 42.7 and Llama 3.2 3B at 61.0. Ministral 8B reaches 8.3 on MTBench and 70.9 on Arena Hard in the same table, while Gemma 2 9B stays ahead on WildBench at 43.8 against 41.3. All of these are Mistral’s own numbers, produced with an internal framework the company says it used to re-evaluate every model for the comparison.

Mistral names on-device translation, smart assistants that work without an internet connection, local analytics and autonomous robotics as the applications customers and partners asked for, citing demand for local, privacy-first inference. The company also positions the two models as intermediaries in front of larger ones such as Mistral Large, tuned to parse input, route tasks and call APIs across multiple contexts in multi-step agent workflows.

Both models went live through Mistral’s API on the day of the announcement, at $0.04 per million tokens for Ministral 3B and $0.1 per million for Ministral 8B, with input and output priced the same. Mistral published weights for Ministral 8B Instruct only, for research use, and lists that model under both the Mistral Research License and the Mistral Commercial License, while Ministral 3B appears under the commercial license alone with no weights release. Anyone wanting to self-deploy either model has to contact Mistral for a commercial license, and the company says it will help with lossless quantisation for specific use cases.