https://llmobile.news/tags/benchmarks/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.news/ticker/artificial-analysis-mobile-benchmark/https://llmobile.news/ticker/artificial-analysis-mobile-benchmark/Mon, 24 Aug 2026 15:00:00 +0200 Chart: Artificial Analysis. Artificial Analysis has published a benchmark of small language models running locally on an iPhone 17 Pro. The study covers 41 quantised builds, 33 of which ran successfully on the device. Models were executed with llama.cpp at 4-bit quantisation or smaller, and had to fit within 8 GB of memory including the KV cache at 8K context. Artificial Analysis states that the inference benchmarking is operated in partnership with Liquid AI. Nanbeige4.2-3B and LFM2.5-2.6B tied for the highest intelligence score at 63. On speed and memory the two differ sharply: LFM2.5-2.6B completed the test prompt in 8.0 seconds using 2.3 GB, against 21.4 seconds and 4.0 GB for Nanbeige4.2-3B. Ornith-1.0-9B scored 62 and Qwen3.5 9B (Reasoning) 61. End-to-end generation time for a 1,024-token prompt plus 256 output tokens ranged from 0.9 seconds for LFM2.5-230M to 26.7 seconds for Falcon-H1R-7B. Peak in-test memory at 4K context ran from 0.4 GB to 6.9 GB. The intelligence index combines tool calling, instruction following, knowledge, scientific reasoning and mathematics. Artificial Analysis notes that context limits, not parameter count alone, held back several 9B models, which frequently hit a 16K ceiling. Chart: Artificial Analysis. Source: https://artificialanalysis.ai/articles/mobile-phone-intelligence-inference Read the article: https://llmobile.news/ticker/artificial-analysis-mobile-benchmark/BenchmarksiPhoneQuantisationllama.cppLiquid AIhttps://llmobile.news/ticker/lfm2-5-2-6b/https://llmobile.news/ticker/lfm2-5-2-6b/Tue, 04 Aug 2026 17:00:00 +0200Liquid AI has released LFM2.5-2.6B, a 2.6-billion-parameter model built on its LFM2 architecture for edge deployment. The company reports 30 tokens per second on a phone, with CPU inference staying under 2.5 GB of memory. Its published chart identifies the device as a Qualcomm Snapdragon Galaxy (SM-S948U1) and puts memory use at 2,448 MB at Q4_K_M quantisation with a 4K-token input context. Chart: Liquid AI. On other CPUs Liquid AI cites 220 tokens per second on an M5 Max and 113 tokens per second on a Ryzen AI Max+ 395. The model was pre-trained on roughly 34 trillion tokens, uses a 128K vocabulary, and its mid-training included a dedicated 128K context-extension phase. Liquid AI positions the model for agentic use, describing planning, tool calling and multi-step tasks, and states that it is competitive with, and often outperforms, models nearly four times its size on instruction-following and tool use, while larger models keep an advantage in coding. Weights are on Hugging Face in BF16, GGUF for llama.cpp, and ONNX. Source: https://www.liquid.ai/blog/lfm2-5-2-6b Read the article: https://llmobile.news/ticker/lfm2-5-2-6b/Liquid AIOpen weightsLFMBenchmarksAgentshttps://llmobile.news/ticker/mlperf-inference-mobile/https://llmobile.news/ticker/mlperf-inference-mobile/Mon, 15 Jun 2026 17:00:00 +0200MLCommons announced MLPerf Mobile v6.0 on June 15, 2026, adding generative AI tests that run Llama on an Android device and report token throughput. The new workload comes in 1B, 3B and 8B parameter sizes, using Llama 3.2 1B Instruct, Llama 3.2 3B Instruct and Llama 3.1 8B Instruct. Each size runs twice, once against TinyMMLU, a small multiple-choice knowledge set, and once against IFEval, which checks whether a model obeys the instructions in a prompt, according to the task configuration in the app repository. MLCommons says the tests run on devices with enough memory through the CPU without tailored acceleration, and that v6.0 adds NPU-accelerated execution of Llama 3.1 8B Instruct on Qualcomm Snapdragon 8 Elite Gen 5. The v6.0 app release of June 9, 2026 also brings support for MediaTek Dimensity 9500 chips and updates support for the Samsung Exynos 2600. The rest of the suite covers image classification with MobileNetV4 on ImageNet, object detection on COCO 2017, segmentation with MOSAIC on ADE20K, language understanding with MobileBERT on SQuAD 1.1, super resolution with EDSR, and Stable Diffusion 1.5 on COCO captions, as the benchmark page lists them. Almost every test runs in the single stream scenario, where the score is the 90th percentile latency for one query at a time, and image classification also runs an offline pass scored as throughput. Stable Diffusion, the first generative test in the suite, arrived in the v4.1 app release of February 11, 2025. Mostafa El-Khamy, co-chair of the MLPerf Mobile working group, had set out the plan in May 2024. Looking ahead, the MLPerf Mobile group is working to add generative AI tasks to a future version of the benchmark suite. Results come from chip and device makers that run the app and submit their logs. The published v6.0 results hold one submission in the closed division, from Samsung on an Exynos 2600 phone, and it covers the vision tests, language understanding, super resolution and Stable Diffusion rather than the new Llama ones. Samsung’s logs give a 90th percentile single stream latency of 0.56 ms for image classification, 0.83 ms for MobileBERT and 1.87 s for one Stable Diffusion image. The v5.0 round carried submissions from Samsung and Qualcomm Innovation Center. Screenshots: Samsung's submission to MLPerf Mobile v6.0, published by MLCommons. Submissions are held back until review closes under the MLCommons submission rules, which keep every result confidential inside a review committee after the deadline and let submitters file objections that the committee then decides. MLCommons publishes the accepted runs in the results table on the MLPerf Inference Mobile page, with the logs and calibration notes in a per-round repository on GitHub. The app is available through Google Play, the Apple App Store and the MLPerf Mobile repository under the Apache 2.0 license. Source: https://mlcommons.org/2026/06/mlperf-mobile-v6/ Read the article: https://llmobile.news/ticker/mlperf-inference-mobile/BenchmarksAndroidLlamaNPUQualcommhttps://llmobile.news/ticker/caped-privacy-defence/https://llmobile.news/ticker/caped-privacy-defence/Wed, 10 Jun 2026 22:48:00 +0200Researchers at the Chinese University of Hong Kong, Southeast University and Shandong University published CAPED on June 10, 2026, a filter that blacks out parts of a phone’s screenshots before they reach a cloud-hosted GUI agent. Across a 28-task privacy suite the authors built themselves, weighted seeded leakage falls from 0.766 with raw screenshots to 0.268 with the filter in front, counted only on runs where the agent still finished its task. A multimodal model plays the attacker and scores what it can still infer from the whole screenshot sequence, not from single frames. A screenshot-based agent operates a phone the way a person does, by looking at the screen, so every image it uploads carries whatever happens to sit next to the thing the user asked for. The authors call this incidental visual privacy exposure and name contacts, message previews, non-target photos, filenames, recommendation feeds and health widgets as content the remote model receives although the user never mentioned it. In one of their test shops, a request to buy sunglasses puts prenatal vitamins, a baby bottle set and a maternity dress in front of the model, because the shop’s recommendation rows share the page with the product. Stripping names and numbers out of the text does not close that gap, and the authors measure text-only anonymisation at 0.752 against 0.766 for no protection at all. Figure: Shen et al., licensed CC BY 4.0. The mock SunShop task asks the agent to buy sunglasses, so the personalised rows around it count as incidental exposure. CAPED decides per interface element rather than per screen, according to the paper. It first sorts the current screen into one of three postures, so galleries, maps, files and financial views show their text but hide their images, while messaging, contacts, feeds and health dashboards hide both by default. The user’s instruction is turned into a short list of what the agent is allowed to need, and a hidden element is released only when it matches that list, checked either by text matching or by asking a vision model whether a cropped thumbnail really shows the requested subject. Hidden areas keep their place in the layout under a solid overlay with an eye icon, and names in exposed text are replaced by placeholders the remote agent can still follow. The defence is meant to run on the phone, and the authors argue the instruction has to be interpreted locally too, since it can name people, relationships, locations and intentions before any screen policy applies. The prototype they measured does not run on a phone. They state that CAPED’s components ran on the evaluation host with an evaluation-side service for the heavier models, and that the timings should be read as live-loop prototype latency rather than physical-phone latency, energy or thermal behaviour. Over 1,394 measured steps the privacy work added 2.325 seconds per agent step, 22.6 percent of a 10.835-second step, most of it the detector at 1.811 seconds, plus 5.716 seconds once per task to extract the requirements. The protection also costs task success. On AndroidWorld’s 116 ordinary tasks, GUI-Owl-7B completed 77 of them (66.4%) unprotected and 64 (55.2%) behind the full filter, which the authors call a prototype-level utility cost. They name further limits themselves, writing that screens their rule base does not cover fall back to the most permissive posture, which they describe as not privacy-conservative, and that open-ended requests such as finding the latest receipt widen exposure and account for much of the leakage that remains. On the connected Gmail, Drive and Photos tasks the filter moves leakage from 0.402 to 0.347 only, the interactive unmasking path was never tested with real users, and the authors write that they plan to release the CAPED artifact in a future public version. Source: https://arxiv.org/abs/2606.12666 Read the article: https://llmobile.news/ticker/caped-privacy-defence/AgentsAndroidPrivacyResearchBenchmarkshttps://llmobile.news/ticker/litert-lm-on-device-genai/https://llmobile.news/ticker/litert-lm-on-device-genai/Tue, 19 May 2026 18:00:00 +0200Google has detailed LiteRT-LM, the runtime it uses to run Gemma 4 on-device in Chrome, ChromeOS, the Pixel Watch and the Google AI Edge Gallery app. It sits on top of LiteRT, formerly TensorFlow Lite, and adds GenAI libraries over the XNNPACK and ML Drift kernels. Running Gemma 4 E2B without multi-token prediction, Google measures 52 decode tokens per second on the Android GPU through OpenCL, 56 on iOS through Metal, and up to 76 in the browser on WebGPU. The Android figures come from a Samsung S26 Ultra, iOS from an iPhone 17 Pro, and web from Chrome on a 2024 MacBook Pro with an M4 Max. Chart: Google. The published comparison puts LiteRT-LM prefill at 3,808 tokens per second against 292 for llama.cpp on the Android GPU, 2,878 against 1,493 for MLX on iOS, and 4,182 against 1,127 for ONNX on the web. Decode figures in the same chart are 52 against 19, 56 against 32 and 76 against 29. Multi-token prediction raises decode further. Google reports up to a 2.2x speedup, taking Gemma 4 E2B from 52 to 85 tokens per second and E4B from 21 to 47, measured on a Samsung S26 Ultra GPU. The drafter and the primary model run on the same hardware block so the shared KV cache and activations stay in local memory. Chart: Google. On memory, LiteRT-LM keeps per-layer embeddings out of memory and loads image and audio encoders only when a task needs them. Google states that the 2.58 GB Gemma 4 E2B model runs with a physical footprint of 607 MB on Apple mobile CPUs using XNNPACK weight caching. For agentic use the runtime supports Thinking Mode, which developers can stream to the interface or strip to save KV cache space: By dedicating a scratchpad for step-by-step reasoning before the model commits to an action, LiteRT-LM can significantly improve the output quality. Constrained decoding enforces JSON schemas on tool payloads, and native function calling pauses execution, returns the tool-call request to the app and resumes on the tool’s output. Session save and restore serialise KV cache state so long conversations skip the prefill phase on return. Beyond the existing Kotlin and C++ interfaces on Android, Google is adding an open-source Swift API for Apple platforms and a JavaScript API for the web through WebAssembly and WebGPU. Source: https://developers.googleblog.com/blazing-fast-on-device-genai-with-litert-lm/ Read the article: https://llmobile.news/ticker/litert-lm-on-device-genai/GoogleLiteRTGemmaBenchmarksAndroidhttps://llmobile.news/ticker/small-vlms-video-compressors/https://llmobile.news/ticker/small-vlms-video-compressors/Thu, 09 Apr 2026 13:40:25 +0200Researchers at Meta AI and KAUST published Tempo on April 9, 2026, a way to hand a long video to a small vision-language model first and let a larger language model answer the question afterwards. On LVBench, a benchmark whose clips average 4101 seconds, the authors report their 6B system scoring 52.7 against 48.2 for VideoChat-Flash 7B, the strongest specialised baseline in their table, and against 30.8 for GPT-4o and 33.1 for Gemini 1.5 Pro. The small model does the reading and the large one does the reasoning. A Qwen3-VL-2B-Instruct compressor takes an eight-frame chunk of video together with the user’s question and writes out a short run of memory tokens that hold what the chunk contains, and a Qwen3 4B language model then reads the whole condensed sequence with timestamps attached and produces the answer. A linear projector joins the two, for 6B parameters in total. The compressor is the part that could in principle sit on a phone, since it never sees more than a few seconds of video at a time. How many tokens each chunk gets is decided during the same forward pass. The authors insert a yes or no question about whether the segment is relevant to the query, read the difference between the two logits as a score, and keep more memory tokens for chunks that score high, spending between 0.5 and 16 tokens per frame under a fixed overall budget of 4K or 8K visual tokens. On LVBench under the 4K budget the measured average comes to 2.9 tokens per frame. The tighter budget also scores better than the looser one on that benchmark, 52.7 against 52.3, and on the long subset of Video-MME, 57.8 against 57.0. Raising the cap to 2048 frames and a 12K budget brings LVBench to 53.7. Diagram: Fei et al., Figure 2 of the paper, licensed CC BY 4.0. On the other three benchmarks the authors report 67.8 on Video-MME under the 4K budget against 65.3 for VideoChat-Flash and 61.9 for the Qwen3-VL 2B model the compressor starts from, plus 75.6 on MLVU and 65.1 on LongVideoBench. All of these are accuracy scores run through lmms-eval, not speed or memory figures. Nothing in the paper was measured on a phone, a tablet or any other edge device. Training used 32 and then 64 Nvidia H100 GPUs, the paper reports no latency, throughput or memory footprint for the compressor on mobile hardware, and the on-device case rests on the size of the model rather than on a measurement. The authors state that all training datasets they used are publicly available, and they link code and a project page from the paper, with the repository carrying an Apache 2.0 license. They describe no weights release in the paper itself. Their stated next steps are to let the compressor decide for itself when it has gathered enough evidence to stop, which they say would bottleneck inference latency as things stand. Source: https://arxiv.org/abs/2604.08120 Read the article: https://llmobile.news/ticker/small-vlms-video-compressors/MetaQwenResearchBenchmarksDistillationhttps://llmobile.news/ticker/llm-inference-at-the-edge/https://llmobile.news/ticker/llm-inference-at-the-edge/Tue, 24 Mar 2026 19:28:00 +0100Four authors at Conscious Engines published a benchmark of Qwen 2.5 1.5B at 4-bit across four edge platforms on March 24, 2026, feeding each device the same 258-token prompt 20 times in a row with a 1-second gap instead of once. On an iPhone 16 Pro the first two runs peak at 40.49 tokens per second, and from run 17 onwards the phone holds a plateau of 23.67 tok/s, which the authors put at 41.5 percent below that peak. The iPhone passes through the three thermal states iOS reports to an app. Runs 1 and 2 sit in Normal and average 36.83 tok/s, runs 3 to 16 sit in Warm and average 24.96 tok/s, and runs 17 to 20 sit in Hot at 23.67 tok/s with a variation of 1.2 percent. The authors write that the 1-second gap allows no thermal recovery, and they measure 5 percent of battery drained over the 20 runs, which they project to roughly 400 replies of this length per charge. A Samsung Galaxy S24 Ultra degrades more gently in the same test. Its Adreno 750 GPU boosts to 1,000 MHz on runs 4 and 5 for a peak of 12.21 tok/s, then steps down to a 720 to 770 MHz plateau from run 8 where the authors measure 10.38 tok/s, 15 percent off the peak, with a low of 9.55 tok/s on the final run. They stress that reaching this plateau took deliberate tuning, with the display off, the context capped at 2,048 tokens and MLC-LLM’s prefill chunk cut to 128 tokens, and that earlier unconstrained runs hit an OS-enforced GPU frequency floor. Neither phone ran the model on its NPU. The paper states that MLX dispatches to the A18 Pro GPU through Metal and does not target Apple’s Neural Engine, and that the S24 Ultra ran through MLC-LLM on the Adreno GPU, leaving the Snapdragon 8 Gen 3’s Hexagon NPU outside the measurement. The one NPU that did run the model, a Hailo-10H module in a Raspberry Pi 5, was also the only part that never throttled, holding 6.914 tok/s across all 20 runs with a variation of 0.04 percent at under 2 W, its CPU at 52.7°C and the NPU at 58.5°C, with attention layers that hailo-ollama does not offload falling back to the Pi’s Cortex-A76 CPU. A user running llama.cpp’s Hexagon NPU backend on a Snapdragon 8 Gen 3 phone reported 12.5 tok/s of generation on Gemma 3 4B and said the phone stayed cool. A laptop RTX 4050 on battery led on speed at 131.7 tok/s and 34.12 W, its GPU climbing from 55°C to 70°C without throttling. Energy per token comes out at 270.5 mJ for the Hailo module measured across the whole system, 297.3 mJ for the RTX 4050 measured at the GPU, and 143.0 mJ for the S24 Ultra plateau derived from the Android fuel gauge with the display off. The authors state that the three readings differ in scope and call the resemblance suggestive rather than settled, and report no energy figure for the iPhone because iOS exposes no per-component power reading. They tested one model, one prompt and one unit per platform across four different runtimes, and describe the results as characterising hardware and software together rather than hardware on its own. Source: https://arxiv.org/abs/2603.23640 Read the article: https://llmobile.news/ticker/llm-inference-at-the-edge/BenchmarksNPUiPhoneSamsungResearchhttps://llmobile.news/ticker/mobilellm-flash/https://llmobile.news/ticker/mobilellm-flash/Mon, 16 Mar 2026 23:10:00 +0100Meta AI published MobileLLM-Flash on March 16, 2026, three models of 350M, 650M and 1.4B parameters whose shapes were picked by measuring how fast each candidate actually ran on a phone. Meta reports the family reaching up to 1.8 times faster prefill and 1.6 times faster decode than Liquid AI’s LFM2 models on mobile CPUs, at comparable or higher accuracy, with support for up to 8k tokens of context. Meta’s first MobileLLM paper argued in 2024 that deep and thin networks win below a billion parameters, and the Flash paper states that such deep-and-thin structures often fail to improve on-device latency. Meta exported 100 architectures to a phone and found parameter count a poor predictor of measured prefill and decode speed, at a Kendall rank correlation of 0.40, with FLOPs little better at 0.46 and 0.55. The 1.4B model therefore runs 16 layers at a width of 2048, against the 30 layers at 1280 of MobileLLM-Pro-1B. Meta ran the latency tests on a Samsung Galaxy S25, a phone with a Snapdragon 8 Elite chipset, an octa-core CPU and 12 GB of memory, exporting each model through ExecuTorch to the XNNPACK CPU backend at 4-bit weights, 8-bit dynamic activations and a quantised KV cache, on 4 CPU threads and averaged over three runs after a warmup. At a 2k context the 650M model reaches its first token in 3.34 s against 6.01 s for LFM2 700M, and decodes at 85.35 tok/s against 53.57 tok/s. Meta repeated the 1k-context test on an iPhone 17, where the 650M model takes 2.81 s to the first token against 3.40 s for LFM2 700M. Most of the speed comes from letting some layers skip attention, so a token passes through those blocks without being compared to every earlier token. The 350M model keeps full attention in 7 of its 12 layers and the 650M model in 8 of 13, while the 1.4B model attends in all 16. Meta reports that skipping consistently beat sliding window attention, which instead limits each token to a fixed span of recent ones, and that more than three skipped layers in a row costs accuracy at the same latency, dropping TriviaQA exact match from 33.2% to 8.8%. Meta built the three models by pruning a shallow 1.8B variant of MobileLLM-Pro and inheriting its weights, which the paper says took about 35% of the tokens that pretraining MobileLLM-Pro required. Across nine reasoning and knowledge tasks Meta reports averages of 45.46, 48.57 and 55.06 for the three sizes, against 44.92, 48.52 and 50.48 for LFM2 at 350M, 700M and 1.2B, 51.79 for Gemma 3 1B and 49.01 for Llama 3.2 1B. After instruction tuning, the 1.4B model scores 47.89 on MMLU and 46.34 on HumanEval, against 49.30 and 37.80 for Llama 3.2 1B. The paper announces no release of the weights. Chart: Meta. Bubble area shows parameter count, latency measured on a Samsung Galaxy S25. Source: https://arxiv.org/abs/2603.15954 Read the article: https://llmobile.news/ticker/mobilellm-flash/MetaResearchBenchmarksAndroidPyTorchhttps://llmobile.news/ticker/qwen-3-5/https://llmobile.news/ticker/qwen-3-5/Mon, 02 Mar 2026 10:00:00 +0100Alibaba’s Qwen team released Qwen3.5-0.8B and Qwen3.5-2B on March 2, 2026, together with 4B and 9B versions, according to the Qwen release notes. Both are dense vision-language models with a 262,144-token context and Apache 2.0 weights. The model card states that the released files work with Hugging Face Transformers, vLLM, SGLang and KTransformers, and that in light of the parameter scale the intended uses are prototyping, task-specific fine-tuning and other research or development purposes. The benchmark table in that model card puts Qwen3.5-2B at 55.3 on MMLU-Pro with reasoning switched off, where the same table gives 40.2 for last generation’s Qwen3-1.7B and 69.6 for the larger Qwen3-4B-2507, and Qwen3.5-0.8B at 29.7. With reasoning on, Alibaba reports 66.5 for the 2B and 42.3 for the 0.8B on the same test, and 51.6 against 11.9 on GPQA, a set of graduate-level science questions. The same table has Qwen3-1.7B ahead of both new sizes on IFEval, which checks whether a model obeys explicit formatting instructions, at 68.2 against 61.2 for the 2B and 52.1 for the 0.8B with reasoning off. Alibaba ran all of these figures itself. Both sizes ship a vision encoder in the same checkpoint rather than as a separate release, and Alibaba’s vision table gives Qwen3.5-2B 64.2 on MMMU, a college-level image question set, above the 61.4 it lists for the previous Qwen3-VL-2B, with 49 for Qwen3.5-0.8B. The two models stack 24 layers in which three linear-attention blocks, whose memory cost does not grow with the sequence, alternate with one full-attention block. Input embedding and output layer are tied, so a single 248,320-entry matrix does both jobs, and Alibaba puts language coverage at 201 languages and dialects. Alibaba published the weights on Hugging Face and ModelScope under Apache 2.0, and its MNN team put 4-bit exports of both sizes up the same day. The 0.8B build carries a 470 MB language weight file plus 63 MB for the vision encoder, and the 2B build 1.18 GB plus 196 MB, against 1.75 GB and 4.55 GB for the full-precision checkpoints. For other local runtimes the Qwen repository points readers to llama.cpp, which it says supports the series for text and vision, and to mlx-lm and mlx-vlm on Apple silicon, without Alibaba publishing GGUF or MLX builds of these sizes itself. Source: https://huggingface.co/Qwen/Qwen3.5-2B Read the article: https://llmobile.news/ticker/qwen-3-5/AlibabaQwenOpen weightsQuantisationBenchmarkshttps://llmobile.news/ticker/nemotron-flash/https://llmobile.news/ticker/nemotron-flash/Mon, 24 Nov 2025 09:46:00 +0100Nvidia published Nemotron-Flash on November 24, 2025, a family of small language models built around measured decoding latency instead of parameter count. Nvidia reports Nemotron-Flash-1B decoding 8k tokens in 14.45 seconds on a single Nvidia H100 at batch size 1, against 27.55 seconds for Qwen3-0.6B, while averaging 49.63 across 16 benchmarks to Qwen3-0.6B’s 44.11. The paper was accepted at NeurIPS 2025. The models mix three kinds of layer rather than using attention throughout. Nvidia interleaves DeltaNet and Mamba2 layers, operators that carry a fixed-size state instead of a cache that grows with the context, with a small number of ordinary attention layers, and an evolutionary search picked which operator goes where. Nemotron-Flash-1B holds 0.96B parameters in 12 blocks with only 2 full attention layers, and Nemotron-Flash-3B holds 2.7B parameters in 18 blocks with 3. Nvidia built both wider and shallower than other models of the same size, where Qwen3-0.6B stacks 28 layers, and writes that deep and thin models are parameter efficient but sit off the accuracy-latency frontier. Nemotron-Flash-3B averages 60.98 across the same 16 tasks, which cover MMLU, commonsense reasoning, math, coding and recall, against 58.96 for Qwen2.5-3B and 55.47 for Qwen3-1.7B. On the same H100, with all models running through TensorRT-LLM AutoDeploy kernels and CUDA Graph, Nvidia measured 28.71 seconds for the 3B against 49.40 for Qwen2.5-3B and 36.20 for Qwen3-1.7B, so 1.7 and 1.3 times lower latency. Throughput was measured separately, with a 32k-token input at the largest batch size that still fits in memory, and Nvidia reports 2939 tok/s for the 3B against 459 and 157 for the two Qwen models, and 7289 tok/s for the 1B against 160 for Qwen3-0.6B. Every figure here is Nvidia’s own, taken on data centre GPUs, and the paper gives no phone or embedded measurements, with the earlier depth and operator studies run on an Nvidia A100. Nvidia trained both models on 4.5T tokens using 256 H100 GPUs and released three checkpoints on Hugging Face, Nemotron-Flash-1B, the 3B base model and an instruction-tuned Nemotron-Flash-3B-Instruct, which averages 52.84 across MMLU, GPQA, GSM8K and IFEval against 48.17 for Qwen2.5-1.5B. The model cards set the licence to Creative Commons Attribution-NonCommercial 4.0, so the weights are downloadable but not licensed for commercial use, and neither Apache nor an Nvidia open model licence applies. Running them needs the TensorRT-LLM AutoDeploy attention kernel alongside the Mamba2 and FlashLinearAttention implementations, since Nvidia states that vLLM and TensorRT-LLM do not support DeltaNet-style linear attention on their own. Chart: Fu et al., Figure 1 of the paper. Nvidia's own measurements on an Nvidia H100. Source: https://arxiv.org/abs/2511.18890 Read the article: https://llmobile.news/ticker/nemotron-flash/NvidiaResearchBenchmarksOpen weightshttps://llmobile.news/ticker/intelligence-per-watt/https://llmobile.news/ticker/intelligence-per-watt/Tue, 11 Nov 2025 07:33:00 +0100Researchers at Stanford University and Together AI profiled more than 20 local models on 1M real queries and report that at least one model with 20B or fewer active parameters answered 88.7% of them correctly. The best single model in their set, GPT-OSS-120B, reached 71.4% on its own, averaged across the four query collections. Their 2025 efficiency figure comes from that model running on an Apple M4 Max. The paper proposes intelligence per watt, mean task accuracy divided by the mean power the accelerator draws while answering, so a pairing that gets more answers right on the same power budget scores higher. The best local pairing each year moved from 7.92×10⁻⁴ accuracy per watt for Mixtral 8x7B on an NVIDIA Quadro RTX 6000 in 2023, to 1.80×10⁻³ for Llama 3.1 8B on an RTX 6000 Ada in 2024, to 4.18×10⁻³ for GPT-OSS-120B on an Apple M4 Max in 2025, which the authors put at a 5.3 times gain over two years. Query coverage for that single best pairing rose from 23.2% to 71.3% across the same three years. A companion metric divides accuracy by the energy a query consumes instead, so slow generation counts against a device. Diagram: Saad-Falcon et al., Figure 1 of the paper, licensed CC BY 4.0. The hardware panel separates the accelerators the authors treat as local from the cloud parts. The 1M queries come from four collections. The team took 500K prompts from WildChat, a month of real ChatGPT traffic, 500K from NaturalReasoning, 12K from MMLU Pro and 26.5K from SuperGPQA, all single-turn. WildChat has no ground truth, so GPT-4o judged whether a local model’s answer matched one written by Qwen3-235B, while the other three collections supplied reference answers. Every run used a batch size of 1, one query at a time. The hardware the authors call local is desktop and workstation class rather than mobile. Their 2025 numbers ran on an Apple Mac Studio with an M4 Max, 128 GB of unified memory and 480 W of continuous power, and the 2023 and 2024 points used NVIDIA workstation cards. No phone, tablet or wearable was profiled. Running the same Qwen3 models, the M4 Max reached 1.40 times lower accuracy per watt than an NVIDIA B200 and 1.6 to 2.3 times lower accuracy per joule, a gap the authors read as headroom for local accelerator design. For the savings case the team simulated 80.2M queries over 24 hours split between four small models on M4 Max machines and Qwen3-235B in the cloud. Perfect routing to the smallest capable model cut energy by 80.4% against sending everything to the frontier model, and a router that picks correctly 80% of the time cut it by 64.3%, with the perfect case described as a theoretical maximum. The authors note that coverage falls in technical domains such as architecture and engineering, that GPT-OSS-120B answered 88.9% of WildChat queries against 64.9% of NaturalReasoning ones, and that they release their profiling harness so the measurements can be repeated on new hardware. Source: https://arxiv.org/abs/2511.07885 Read the article: https://llmobile.news/ticker/intelligence-per-watt/ResearchBenchmarksApple SiliconNvidiaQwenhttps://llmobile.news/ticker/lm-meter/https://llmobile.news/ticker/lm-meter/Tue, 07 Oct 2025 19:05:00 +0200Researchers at Georgia State University and Toyota InfoTech Labs published lm-Meter, a latency profiler that runs inside the inference engine on the phone and splits each generation into embedding, prefill, decode, softmax and sampling. Measuring the Pythia models on a Google Pixel 8 Pro, they report that scaling from 70M to 1.4B parameters raises prefill latency from 0.012 s to 1.9 s per input token, a 158x slowdown, while decode latency per output token grows from 0.015 s to 0.15 s, a 10x slowdown. The authors write that this inverts the server picture, where decode is usually the limiting phase for single-request inference. Below the phase level the profiler times individual GPU kernels through OpenCL event timestamps, which give queue, submit, start and end times without access to the closed-source driver. Running a 4-bit quantised Gemma-2-2B-it on a Pixel 8 Pro, the authors report that fused matrix-multiplication kernels dominate a decode step and that the GPU sits idle for more than 21% of it, the second-largest contributor to the step, which they attribute to host-side data preparation and I/O stalls. The paged attention kernel that scans the growing key-value cache is the only one whose cost rises with position in the sequence, climbing from roughly 0.2 ms to about 0.8 ms per token over 250 decode steps, and idle time drops from about 21% to 12% when the model generates 256 tokens instead of 16. Whether those measurements mean anything depends on what the profiler itself costs. lm-Meter sits in the MLC LLM runtime and TVM in about 3,500 lines of code and needs no host machine attached, and under the Powersave CPU governor, the most constrained setting they tested, the authors measure a throughput loss of 2.58% in prefill and 0.99% in decode. They put the same figures for MELTing Point, the on-device profiler they compare against, at 22% for prefill and more than 93% for decode. Checked against traces from Android GPU Inspector, they report end-to-end phase accuracy of at least 99.99% and mean kernel-level accuracy of 96.82% on the Pixel 8 Pro and 96.61% on a Pixel 7. The code is on GitHub under the MIT license, with the MLC LLM path released for Android GPUs through OpenCL and support for llama.cpp, vLLM, iOS Metal and Nvidia Jetson listed as unfinished. The work was accepted to the ACM/IEEE Symposium on Edge Computing 2025 and funded by Toyota Motor North America. The measurements come from three phones, the Pixel 8 Pro, Pixel 7 and Pixel 6, and the authors state that other edge platforms such as Jetson boards and Intel NPUs may show different bottlenecks. Source: https://arxiv.org/abs/2510.06126 Read the article: https://llmobile.news/ticker/lm-meter/ResearchBenchmarksDeveloper toolsMLC LLMPixelhttps://llmobile.news/ticker/mobilellm-r1/https://llmobile.news/ticker/mobilellm-r1/Fri, 12 Sep 2025 12:00:00 +0200Meta released MobileLLM-R1 on September 12, 2025, a family of reasoning models at 140M, 360M and 950M parameters that write out a chain of intermediate steps before giving an answer instead of replying directly. Meta reports the 950M model scoring 74.0 on the MATH500 maths benchmark and 19.9 on LiveCodeBench v6, against 73.0 and 14.9 for Qwen3-0.6B. The technical report followed on September 29, 2025. The three models saw 4.2T training tokens, which Meta puts at 11.7% of the 36T tokens behind Qwen3-0.6B. Pre-training ran in two phases of 2T tokens each, resampled from a curated pool of roughly 2T tokens of open datasets, followed by two mid-training phases of 100B tokens that distil from Llama-3.1-8B-Instruct as a teacher, then supervised fine-tuning on reasoning traces. Meta uses the result to argue that reasoning does not require the corpora above 10T tokens that recent models have been trained on. For the post-trained models Meta’s table puts MobileLLM-R1-950M at 74.0 on MATH500, 67.5 on GSM8K, 15.5 on AIME 2024 and 19.9 on LiveCodeBench v6, where Qwen3-0.6B reaches 73.0, 79.2, 11.3 and 14.9 and DeepSeek-R1-Distill-Qwen-1.5B reaches 83.2, 77.3, 29.1 and 19.9. Among the models whose weights, data and recipe are all public, the same AIME figure compares with 0.6 for OLMo-2 1.48B and 0.3 for SmolLM2-1.7B. On code the 950M base model scores 46.3 on HumanEval against 30.5 for Qwen3-0.6B-Base, and the 140M base model reaches 15.9 where SmolLM2-135M scores 0.0. Meta profiled the models through ExecuTorch on a Samsung Galaxy S22 with 8 GB of RAM, quantised to 4-bit weights with 8-bit dynamic activations. The 140M model generated 129.67 tok/s at a 1k context and 79.71 tok/s at 32k, the 950M model 31.05 tok/s at 1k before running out of memory at 16k, and Llama-3.2-1B 28.71 tok/s at 1k with the same memory failure. The architecture follows Meta’s earlier MobileLLM work and Llama 3.2, sharing one weight matrix between the input embedding and the output layer, and the post-trained models take a 32k context. Meta released the weights on Hugging Face together with the training code, the data sources and the data mixing ratios on GitHub, under the FAIR Noncommercial Research License, which covers research, education and analysis but not commercial use. Access to the model repositories is gated behind a form. Update, November 24, 2025. Meta released MobileLLM-R1.5 at the same three sizes, fine-tuned from the R1 models with one further round of on-policy distillation, where the student answers first and the teacher, here Llama-3.1-Nemotron-Nano-4B-v1.1, grades its own words back to it token by token. Meta reports the 950M version at 86.6 on MATH500, 82.6 on GSM8K, 39.9 on AIME 2024 and 29.1 on LiveCodeBench v6, up from 74.0, 67.5, 15.5 and 19.9 for MobileLLM-R1-950M, and says it beats DeepSeek-R1-Distill-Qwen-1.5B on every maths and coding benchmark it ran. At 360M Meta puts MATH500 at 63.4 against 28.4 and GSM8K at 52.8 against 24.5. Source: https://arxiv.org/abs/2509.24945 Read the article: https://llmobile.news/ticker/mobilellm-r1/MetaResearchBenchmarksDistillationAndroidhttps://llmobile.news/ticker/pd-device-cloud-split/https://llmobile.news/ticker/pd-device-cloud-split/Tue, 12 Aug 2025 17:56:29 +0200Thirty researchers at Huawei Technologies published P/D-Device on August 12, 2025, a scheme that runs a single language model request across a cloud data centre and a phone at the same time instead of choosing one of them. The cloud reads the prompt and writes only the first few tokens, then the device carries the answer to the end. The authors measure average time to first token falling by at least 60 percent against running the whole request on the device, and cloud throughput rising by 1.6 to 15 times, 7.6 times on average, against letting the cloud generate a full 200-token answer. The split follows from the two phases of inference wanting different hardware. Prefill, where the model reads the prompt before it writes anything, is compute-bound, and the authors report on-device time to first token climbing steeply with prompt length, into seconds or tens of seconds once prompts run to several thousand tokens. Decoding, where the model emits one token at a time, behaves differently, because a phone serves one user while a cloud instance serves a large batch, so the authors find the two roughly matched at tens of milliseconds per output token. Decoding is also what occupies a cloud instance for tens of seconds per request, so the scheme cuts the cloud off after a set number of tokens and hands the rest of the answer to the device. The device displays the cloud’s first token the moment it arrives, so the wait the user actually perceives is the cloud’s prefill, which the authors put at hundreds of milliseconds. Behind that display the device runs its own prefill while a speed controller releases the next tokens from the cloud at a deliberately slowed pace, matched to human reading speed, until the device has caught up and generates the rest itself. During its prefill the cloud also scores the prompt using the attention weights it has already computed, picks the sentences that matter most, and returns the selection as a compressed mask so the device prefills a shorter prompt, with a quarter of the prompt kept by default. What crosses the network is text, never model state. The device sends the prompt as raw text and the cloud returns the first token, the mask and a token budget, which the authors cap at hundreds of kilobytes for an 8k prompt, while the KVCache, the gigabyte-scale intermediate state that other cloud-device schemes ship back and forth, is too large to move in time and is never transferred at all. On the prompt text itself the authors state that the request goes out only under user and app authorisation and after sensitive information such as phone numbers and real names has been removed or substituted, and they cite Apple’s Private Cloud Compute as the class of guarantee they have in mind, without describing an implementation or measuring one. The archive’s post on Flower Intelligence covers the reverse arrangement, where the device runs first and the remote service stays off unless the app turns it on. Diagram: Jin et al. The cloud scores the prompt during its own prefill and returns the selection as a mask alongside the first token. The prototype runs cloud inference on Huawei Ascend NPUs and device inference on Kirin SoCs, tested on a phone and a tablet, with every chip and model anonymised as Type 1, Type 2, Model 1 and Model 2 throughout the paper. On LongBench with the same model on both sides, the authors’ own table gives their refinement an average of 26.64 on single-document QA against 25.89 for the unrefined on-device baseline, and 26.83 against 22.96 on multi-document QA. The authors name the costs themselves, a quality score around 85 percent of cloud-only inference when the cloud runs the larger model, refinement adding one or two hundred milliseconds at an 8k prompt and several hundred under load, and the smoothed display running slightly above their own tolerable threshold of about a hundred milliseconds per token once prompts pass 8k. The paper, on arXiv under a CC BY 4.0 license, further states that connections were tested over Wi-Fi with an IP whitelist rather than LTE, that collaboration across different NPU types is still being explored, and that the device supports prompts of several thousand tokens against hundreds of thousands in the cloud. Source: https://arxiv.org/abs/2508.09035 Read the article: https://llmobile.news/ticker/pd-device-cloud-split/ResearchHuaweiNPUBenchmarkshttps://llmobile.news/ticker/demystifying-slms-edge/https://llmobile.news/ticker/demystifying-slms-edge/Sun, 27 Jul 2025 09:00:00 +0200Ten researchers at Beijing University of Posts and Telecommunications, Peng Cheng Laboratory, the University of Cambridge, Xiaomi, Helixon Research and Flower Labs collected 68 small language models between 100M and 5B parameters, released by 24 organisations between OPT in May 2022 and Llama 3.2 in September 2024, and measured both their accuracy and their cost on edge hardware. In Demystifying Small Language Models for Edge Deployment, presented at ACL 2025, Microsoft’s Phi-3 and Phi-3.5-mini sit highest on the capability chart at about 70 percent average accuracy over 10 reasoning and knowledge datasets, level with the Llama 3.1 line the authors draw for comparison. The runtime half of the study covers the 20 models that llama.cpp supports, quantised to 4-bit and run with a 50-token prompt and 50 generated tokens. Chart: Lu et al., licensed CC BY 4.0. Accuracy is averaged over the 10 datasets with math excluded, and circle size is proportional to model size. Parameter count predicts on-device latency poorly. The measurements put OpenELM-3B at 9.4 percent more parameters than Phi-2 but 9.9 percent faster to the first token, and Qwen1.5-0.5B at 25.4 percent more parameters than Qwen2-0.5B yet 31.9 percent faster to the first token and 11.8 percent faster per decoded token. Qwen1.5-1.8B carries 30.5 percent more parameters than Pythia-1.4B and still reaches the first token 18.5 percent sooner. The authors trace the gap to how a model spends its parameter budget, with Qwen2 sharing its input embedding and output layer and putting the saved weights into wider attention and feed-forward layers, which costs compute in the prefill stage. Memory ran from 275 MB to 2456 MB across the 20 models, measured with llama.cpp on a Jetson Orin NX at a fixed 2048-token context. Vocabulary size moves that figure as much as parameter count does, and the paper reports Bloom-560M needing 492 MB of compute buffer for its 250,880-token vocabulary, 3.5 times what OpenELM-1.1B needs with 32,000 tokens. OpenELM-1.1B uses 23.8 percent less memory than Bloom-560M despite 32.3 percent more parameters, and OpenELM-3B 13.8 percent less than Gemma-2B despite 21.1 percent more, which the authors put down to the small vocabulary and to grouped-query attention, where several attention heads share one set of cached keys and values. Raising each model to its own maximum context window pushes Qwen2-0.5B past 6 GB, with compute buffer and KV cache making up 85 percent of the total for the Qwen2 models. Charts: Lu et al., licensed CC BY 4.0. Measured with llama.cpp at 4-bit, and memory taken on a Jetson Orin NX at a 2048-token context. Quantisation paid off in decoding rather than in prefill. Testing five settings on Phi-1.5, the authors found Q4_K_M the strongest, roughly halving per-token decode latency against FP16, while at a 50-token prompt it cut the time to the first token by only 13 percent. Q6_K and Q3_K matched or exceeded FP16 on longer prompts, which the paper puts down to the alignment and padding their irregular bit widths force. The hardware gap was wider than the quantisation gap, with the Jetson’s GPU 40 times faster than the Meizu 18 Pro’s CPU at prefill but only 1.84 times faster at decode, and the Xiaomi 12S on a Snapdragon 8 Gen 1+ the fastest of the three phones, ahead of the Pixel 7 Pro and the Meizu on its Snapdragon 888. The authors name two limits on their own work. They left math datasets out of the capability evaluation because of the gap between small models and larger ones on mathematical reasoning, and they confined the cost analysis to the 20 models llama.cpp supports so that the inference engine stayed constant, excluding every model it cannot load. The paper is free to read at the ACL Anthology under a Creative Commons Attribution 4.0 licence, and the same group had earlier measured 22 models from 0.5B to 7B on four Android phones, three of which reappear in this test bed. PalmBench adds power draw and surface temperature on comparable hardware, and Artificial Analysis has since reported quality, speed and memory together for 33 quantised models on one iPhone 17 Pro. Source: https://aclanthology.org/2025.acl-long.718/ Read the article: https://llmobile.news/ticker/demystifying-slms-edge/BenchmarksResearchllama.cppMemoryAndroidhttps://llmobile.news/ticker/apple-foundation-models-report-2025/https://llmobile.news/ticker/apple-foundation-models-report-2025/Fri, 18 Jul 2025 01:37:00 +0200Apple posted its 2025 foundation models tech report to arXiv on July 17, 2025, giving the numbers behind the roughly 3-billion-parameter on-device model it shipped at WWDC25. The report’s Table 3 puts the price of compressing that model to 2 bits per weight at 3.4 points of MMLU, from 67.8 in 16 bits down to 64.4, with IFEval falling from 85.1 to 82.3. Apple reaches 2 bits through quantisation-aware training with a learnable scaling factor per weight tensor, and reports that a balanced set of four levels at -1.5, -0.5, 0.5 and 1.5 trains more smoothly than the usual -2, -1, 0 and 1. The embedding table is quantised to 4 bits and the key-value cache to 8 bits, and low-rank adapters are fine-tuned afterwards to claw back the quality the compression costs. The server model takes a different route, compressed after training to 3.56 bits per weight with Adaptive Scalable Texture Compression, a GPU texture format whose fixed-function decode hardware on Apple GPUs unpacks the weights at no cost to the compute cores. The report also spells out the KV cache sharing the framework announcement only named. Apple splits the transformer so that Block 1 holds 62.5 percent of the layers and Block 2 the remaining 37.5 percent with its key and value projections removed, reusing Block 1’s cache instead. That cuts KV cache memory by 37.5 percent, and because Block 2 produces no keys or values, prefill skips its computation entirely, which Apple says cuts time to first token by about the same 37.5 percent. Apple rebuilt how the on-device model is trained. The company first trained a dense model on about 14T tokens, sparse-upcycled it into a 64-expert mixture-of-experts version, where only part of the model runs per token, using 1T high-quality tokens, then retrained the dense model on its last 10 percent of tokens against a distillation loss from that MoE teacher. Apple states this cut the cost of training the teacher by 90 percent and removed the need for structural pruning. The server model was trained on 8192 v5p Cloud TPU chips for 13.4T tokens. For the language expansion the report states the models are designed to support 16 languages, built on a text tokenizer grown from a 100k to a 150k vocabulary and a continued pre-training mix whose multilingual share rose from 8 percent to 30 percent. In Apple’s own human evaluation on US English text prompts, graders preferred the on-device model over Qwen-2.5-3B on 35.3 percent of prompts against 12.9 percent the other way, while against Gemma-3-4B the split was 21.0 percent to 21.9 percent. The report is on arXiv under a Creative Commons Attribution 4.0 license. Chart: Apple's own human evaluation, from the tech report. Rows marked with an asterisk were tested against the compressed on-device model. Source: https://arxiv.org/abs/2507.13575 Read the article: https://llmobile.news/ticker/apple-foundation-models-report-2025/AppleApple SiliconQuantisationDistillationBenchmarkshttps://llmobile.news/ticker/lfm2/https://llmobile.news/ticker/lfm2/Thu, 10 Jul 2025 14:00:00 +0200Liquid AI released LFM2 on July 10, 2025, three open-weight checkpoints of 350M, 700M and 1.2B parameters aimed at the CPUs inside phones, laptops and vehicles. The company reports 2x faster decode and prefill speed on CPU than Qwen3, and describes the models as built for millisecond latency and offline operation on the device rather than in a data centre. The architecture is a hybrid of 16 blocks, 10 of them gated short convolutions and 6 grouped query attention. A short convolution mixes each token with a handful of its neighbours instead of attending over the whole history, which is cheap on a phone CPU and keeps the attention cache small, while the six attention blocks preserve the recall that pure convolutions lose. Liquid AI says the layout came out of its STAR architecture search, which it ran against measured peak memory and measured prefill plus decode speed on Qualcomm Snapdragon embedded SoC CPUs rather than against proxy metrics. Liquid AI benchmarked the models on a Samsung Galaxy S24 Ultra with a Qualcomm Snapdragon SoC and on an AMD Ryzen HX370 laptop, using Q4_0 in llama.cpp and 8da4w in ExecuTorch. In its llama.cpp charts LFM2-350M clears 1,300 tokens per second of prefill and runs at roughly 190 tokens per second of decode on the Galaxy S24 at a 128-token sequence, falling to around 120 tokens per second of decode at 4,096 tokens. The company states that LFM2-700M stays faster than Qwen3-0.6B on both decode and prefill in both runtimes despite being 16 percent larger, and that the CPU gains should carry over to GPUs and NPUs once kernels are optimised. Charts: Liquid AI's own llama.cpp measurements at Q4_0 quantisation. The models are small enough to ship whole, with Liquid AI’s own Q4_0 GGUF builds at 219 MB for LFM2-350M and 696 MB for LFM2-1.2B. On Liquid AI’s internal evaluation table LFM2-1.2B scores 55.23 on MMLU and 74.89 on IFEval against 59.11 and 73.98 for Qwen3-1.7B, a model with 47 percent more parameters, while LFM2-700M reaches 49.9 on MMLU against 40.08 for Gemma 3 1B IT. Liquid AI notes that it evaluated Qwen3 in non-reasoning mode only, on the grounds that reasoning traces run past the output budgets that matter on edge devices. Chart: Liquid AI. The average covers MMLU, IFEval, IFBench, GSM8K and MMMLU. Weights are on Hugging Face, with the models also reachable through the Liquid Playground and OpenRouter, and Liquid AI points developers at llama.cpp and ExecuTorch for local testing. The licence is an open licence based on Apache 2.0 that permits academic and research use freely and commercial use by companies under $10 million in revenue, with anyone above that threshold directed to contact the company for a commercial licence. All three checkpoints were trained on 10T tokens, roughly 75 percent English, 20 percent multilingual and 5 percent code, with the context extended to 32K during pre-training. Update, September 23, 2025. Liquid AI added LFM2-2.6B, which widens the stack to 30 blocks, 22 of them convolutions and 8 attention. The company reports 82.41 percent on GSM8K and 79.56 percent on IFEval, and says the model outperforms Llama 3.2-3B-Instruct, Gemma 3 4B IT and SmolLM3-3B. It ships on Hugging Face under the same licence. Update, November 28, 2025. Liquid AI published an LFM2 technical report covering the dense 350M to 2.6B models plus an 8.3B mixture-of-experts variant with 1.5B active parameters, where only part of the model runs per token. Measuring on a Samsung Galaxy S25 with a Snapdragon 8 Elite in llama.cpp at Q4_0, it reports LFM2-350M at 1,067 tokens per second of prefill and 194.1 tokens per second of decode on a 1K-token prompt, and LFM2-2.6B at 143 and 33.8 against 57 and 17.2 for Qwen3-4B. The report names ExecuTorch, llama.cpp and vLLM as the shipped deployment packages. Source: https://www.liquid.ai/blog/liquid-foundation-models-v2-our-second-series-of-generative-ai-models Read the article: https://llmobile.news/ticker/lfm2/Liquid AILFMOpen weightsllama.cppBenchmarkshttps://llmobile.news/ticker/smollm3/https://llmobile.news/ticker/smollm3/Tue, 08 Jul 2025 12:30:00 +0200Hugging Face released SmolLM3 on July 8, 2025, a 3B model trained on 11.2 trillion tokens that takes up to 128k tokens of context, covers six languages and answers in either of two modes. It is the successor to SmolLM2, whose largest model was 1.7B with an 8k context window and, in Hugging Face’s own words, primarily understands and generates content in English. The longer context comes from an architecture change plus a separate training stage. Hugging Face removed rotary position embeddings from every fourth layer, a technique it calls NoPE, and replaced multi-head attention with grouped-query attention to shrink the key-value cache during inference. After the main pretraining run it trained the model on another 100 billion tokens in two steps, 4k to 32k and then 32k to 64k, and YaRN, a method for extrapolating past the trained length, carries it to 128k tokens at inference, though the published configuration ships set to 64k and has to be edited to go further. Multilingual text made up 12 percent of the web data at every pretraining stage, and Hugging Face lists English, French, Spanish, German, Italian and Portuguese as supported. Hugging Face’s own base-model figures put SmolLM3 at 76.2 on HellaSwag, 65.6 on ARC and 79.0 on BoolQ, ahead of both Llama 3.2 3B and Qwen2.5 3B and ahead of the larger Qwen3 4B on all three. Qwen3 4B stays in front on maths and code, at 51.2 against 46.1 on MATH and 54.9 against 30.5 on HumanEval+, and on the Ruler 64k long-context test Llama 3.2 3B leads at 72.9 against 67.9 for SmolLM3. On the Flores-200 translation benchmark SmolLM3 scores 56.3 against 55.8 for Qwen3 4B and 54.9 for Qwen2.5 3B, and its multilingual chart shows it ahead of both 3B rivals in all five non-English languages and behind Qwen3 4B in all five. Table: Hugging Face's own base-model figures. A flag in the system prompt, /think or /no_think, decides whether the instruct model reasons before it answers. Hugging Face reports the reasoning mode at 36.7 percent on the AIME 2025 maths competition against 9.3 percent without it, 41.7 against 35.7 on GPQA Diamond and 30.5 against 15.2 on LiveCodeBench v4. Qwen3 4B leads on most of the eight tests in that comparison in both modes, at 58.8 on AIME 2025 and 52.9 on LiveCodeBench, though SmolLM3 without reasoning tops it on the IFEval instruction-following test at 76.7 to 68.9. Table: Hugging Face's own instruct-model figures. Hugging Face published the recipe as well as the weights. Pretraining ran in three stages that moved web data from 85 percent to 75 and then 63 percent while code rose from 12 percent to 24 and maths from 3 percent to 13, each share drawn from named public datasets, and the company released the training configs with the exact data weights along with intermediate checkpoints. Both the base and instruct models are on Hugging Face under the Apache 2.0 license, and the model card names llama.cpp, ONNX, MLX, MLC and ExecuTorch for local inference. The 4-bit GGUF published by ggml-org on release day is 1.92 GB, against 3.27 GB for the 8-bit build for MLX and 6.15 GB for the bfloat16 weights. Diagram: Hugging Face. Source: https://huggingface.co/blog/smollm3 Read the article: https://llmobile.news/ticker/smollm3/Hugging FaceOpen weightsBenchmarksOpen sourcellama.cpphttps://llmobile.news/ticker/qwen-3/https://llmobile.news/ticker/qwen-3/Wed, 14 May 2025 15:41:00 +0200Alibaba’s Qwen Team published the Qwen3 Technical Report on May 14, 2025, covering six dense models from 0.6B to 32B parameters and two mixture-of-experts models, where only part of the model is active per token. The two smallest dense sizes, Qwen3-0.6B and Qwen3-1.7B, carry the same mode switch as the rest of the family, so one set of weights either works through a problem step by step before answering or replies straight away. Alibaba reports Qwen3-1.7B scoring 93.4 on MATH-500, a 500-problem maths set, with reasoning on, against 73.0 with it off. Earlier Qwen releases needed two separate models for that, a chat model and a dedicated reasoning model such as QwQ-32B. In Qwen3 the user picks the behaviour with a flag in the prompt or the system message, /think or /no_think, and the report states that reasoning is the default and that across a multi-turn conversation the model follows the last flag it saw. The same section describes a reasoning budget, where the model’s working is cut off once it passes a token limit the user set and it answers from what it has so far, which the authors say they never trained for and which emerged from merging the two modes. The benchmark tables give both small sizes in both modes. For Qwen3-0.6B, Alibaba reports 55.6 on MMLU-Redux, a corrected version of the MMLU knowledge test, with reasoning on against 44.6 with it off, and 77.6 against 55.2 on MATH-500. For Qwen3-1.7B the same pair of tables gives 73.9 and 64.4 on MMLU-Redux, 48.3 against 13.4 on the AIME 2024 competition maths set, and 33.2 against 11.6 on LiveCodeBench v5. Against the reasoning baselines in the report, Qwen3-0.6B is ahead of DeepSeek-R1-Distill-Qwen-1.5B on MMLU-Redux, 55.6 to 45.4, and behind it on MATH-500, 77.6 to 83.9. Alibaba built the small sizes by copying behaviour from the large ones rather than by running the full reinforcement learning pipeline on each. The report describes two stages of distillation from Qwen3-32B and Qwen3-235B-A22B, first on stored teacher outputs generated in both modes, then by matching the small model’s output probabilities against the teacher’s while the small model generates, and it credits the first stage with teaching the smaller models to switch modes at all. Pre-training ran on about 36 trillion tokens covering 119 languages and dialects, up from 29 in Qwen2.5. Context length is where the two small sizes sit apart from the family. Table 1 of the report puts Qwen3-0.6B and Qwen3-1.7B at 32K tokens against 128K for Qwen3-8B and every larger model, and both tie the input embedding to the output layer so one weight matrix does both jobs. All eight models are published under the Apache 2.0 license on Hugging Face, ModelScope and Kaggle. The Qwen3 blog post of April 29, 2025 names Ollama, LM Studio, MLX, llama.cpp and KTransformers as the tools it recommends for running them locally. Source: https://arxiv.org/abs/2505.09388 Read the article: https://llmobile.news/ticker/qwen-3/AlibabaQwenOpen weightsResearchBenchmarkshttps://llmobile.news/ticker/scaling-on-device-gpu-inference/https://llmobile.news/ticker/scaling-on-device-gpu-inference/Thu, 01 May 2025 02:44:00 +0200Eight researchers at Google published ML Drift on May 1, 2025, a GPU inference framework for running large generative models on the graphics chip of a phone, a laptop or a Mac. On the Adreno 750 GPU of a Samsung S24, the authors measure 1370 tokens per second of prefill and 37.1 tokens per second of decode for Gemma2 2B, and 412 and 12.7 tokens per second for Llama 3.1 8B. They report a 5 times to 11 times prefill speedup over llama.cpp, MLC LLM and the Qualcomm AI Hub benchmark on Qualcomm GPUs, and the work is due at the CVPR 2025 Workshop on Efficient and On-Device Generation (EDGE). The team benchmarked five mobile GPUs, the Adreno 830 in a Xiaomi 15 Pro with 16 GB of RAM, the Adreno 750 in a Samsung S24 with 8 GB, the Adreno 740 in a Samsung S23 Ultra with 8 GB, the Arm Immortalis-G720 in a vivo X100 Pro with 16 GB and the Arm Mali-G715 in a Pixel 9 with 12 GB, each at a fixed 1024 prefill and 256 generated tokens. Llama 3.1 8B with every weight at 8-bit ran out of memory on the S24, the S23 Ultra and the Pixel 9 and finished only on the two 16 GB phones, at 7.70 and 4.72 tokens per second of decode. A mixed scheme that holds attention weights at 8-bit and drops embedding and feed-forward weights to 4-bit ran the 8B model on all five devices and lifted decode by up to 1.9 times. ML Drift also treats the two phases of generation separately, since filling the prompt is limited by arithmetic while producing each further token is limited by memory bandwidth. Chart: Google. Solid bars are prefill on the left axis, cross-hatched bars decode on the right axis. Gaps mark runs the authors could not measure. A mobile GPU constrains inference in ways the paper works through one at a time. Memory comes first, and the authors report that Stable Diffusion 1.4 would need 4.31 GB for its intermediate 16-bit activations, which handing the same buffers to operations whose lifetimes do not overlap cuts to 387 MB. Weight layout comes second, and rearranging weight tensors into slices of four channels that match the 4-element units a GPU computes with buys up to a 20% speedup in matrix multiplication, which the authors call the primary driver of the framework’s advantage. Shader code comes third, and ML Drift writes its kernels once at startup from hand-tuned templates, so the translation between a tensor’s logical indices and the buffer or texture that actually holds it costs nothing while tokens are being produced. The same engine renders Stable Diffusion 1.4 at 512 by 512 pixels over 20 iterations in 10.96 seconds on the S23 Ultra, under 9 seconds on the S24 and in 3.4 seconds on a laptop with an Intel Lunar Lake Ultra 7 258V. On an NVIDIA RTX 4090 the OpenCL path runs 5% to 25% slower than llama.cpp on CUDA, which the authors put down to NVIDIA’s Tensor Cores being unreachable through OpenCL and WebGPU. The paper names sub-channel quantisation, sparsity and vendor extensions for matrix multiplication as the next steps, and points to no public release of the framework. Source: https://arxiv.org/abs/2505.00232 Read the article: https://llmobile.news/ticker/scaling-on-device-gpu-inference/GoogleQualcommAndroidBenchmarksResearchhttps://llmobile.news/ticker/gemma-3/https://llmobile.news/ticker/gemma-3/Wed, 12 Mar 2025 17:00:00 +0100Google released Gemma 3 on March 12, 2025 in four sizes, 1B, 4B, 12B and 27B, and describes them as designed to run fast directly on devices, from phones and laptops to workstations. The 1B is a new size for the line, which started at 2B with Gemma and stayed there through Gemma 2. The technical report puts the 1B at 2.0 GB of weights in bfloat16 and at 0.5 GB as a quantised int4 checkpoint, and the 4B at 8.0 GB and 2.6 GB. Context length is not the same across the four. The report gives 128K tokens for the 4B, 12B and 27B and 32K for the 1B, with the three larger models pre-trained at 32K and stretched to 128K at the end of pre-training by rescaling their rotary position embeddings. To stop the key-value cache from swallowing memory at long context, Gemma 3 places five local attention layers between each global one and holds the local layers to a 1024-token window, where Gemma 2 alternated one for one. Google reports that in an ablation on a 2B model a global-attention-only configuration carries a 60 percent memory overhead from the cache at a 32K prefill, and that interleaving with a 1024-token window brings that under 15 percent. Multimodality starts at 4B. The 4B, 12B and 27B share one 400M-parameter SigLIP vision encoder, frozen during training, which takes square images resized to 896 by 896 pixels and condenses each one into 256 tokens for the language model to read. An inference-time step Google calls Pan and Scan cuts non-square or high-resolution images into equal crops and resizes each to the encoder’s resolution, which the report measures as raising the 4B from 72.8 to 81.0 on DocVQA and from 44.1 to 57.0 on InfoVQA in a 4-shot evaluation of a pre-trained checkpoint. The 1B has no vision encoder and takes text only. Google published quantisation-aware trained checkpoints next to the raw weights, made by fine-tuning each model for about 5,000 steps against the probabilities of the unquantised model, so the weights are trained to tolerate the lower precision rather than rounded down afterwards. The three formats are per-channel int4, per-block int4 and switched fp8, which Google picked for what open source inference engines such as llama.cpp already support. The report’s footprint table counts weights alone and weights plus a key-value cache for a 32,768-token sequence, and on the second measure the 1B needs 1.4 GB at per-channel int4 against 2.9 GB in bfloat16, while the 4B needs 7.3 GB against 12.7 GB. Google's own footprint figures for the raw and quantised checkpoints, with and without a key-value cache at 32,768 tokens. Table: Google DeepMind, Gemma 3 technical report, CC BY 4.0. Google reports its own zero-shot figures for the instruction-tuned models and puts Gemma 3 4B at 43.6 on MMLU-Pro and 75.6 on MATH, against 15.6 and 27.2 for Gemma 2 2B and 56.9 and 55.6 for Gemma 2 27B, with the 1B at 14.7 and 48.0. The 4B also scores 48.8 on the MMMU validation set, a benchmark the text-only sizes do not run, and the report calls the instruction-tuned 4B competitive with the instruction-tuned Gemma 2 27B. Google put every size on Hugging Face, Kaggle and Ollama as a pre-trained and an instruction-tuned checkpoint under the Gemma Terms of Use, and names LiteRT, Gemma.cpp, llama.cpp and Google AI Edge among the runtimes that take them. The same release added ShieldGemma 2, a 4B image safety classifier built on Gemma 3 that labels images for dangerous content, sexually explicit material and violence. ▶What's new in Gemma 3?Loading connects your browser to www.youtube-nocookie.com, which may process your IP address and use cookies. Load contentOpen externallyVideo: Google for Developers. Source: https://blog.google/technology/developers/gemma-3/ Read the article: https://llmobile.news/ticker/gemma-3/GoogleGemmaQuantisationOpen weightsBenchmarkshttps://llmobile.news/ticker/on-device-inference-tradeoffs/https://llmobile.news/ticker/on-device-inference-tradeoffs/Wed, 12 Mar 2025 08:01:00 +0100Maximilian Abstreiter, Sasu Tarkoma and Roberto Morabito of the University of Helsinki and EURECOM measured 11 language models from 0.5B to 9.2B parameters on a Raspberry Pi 5 and an NVIDIA Jetson Orin Nano Developer Kit, both with 8 GB of RAM, running every model through llama.cpp at 4-bit. On the Pi, the authors report that only models up to Llama 3.2 3B, or up to Phi 3.5 mini in the faster of the two 4-bit formats they tested, generated text above 5.3 tokens per second, the rate they derive from average English reading speed. The Orin’s GPU stayed above that rate for all 11 models, its CPU cores up to InternLM 2.5 7B in that same format. Generation speed drops off steeply with model size in the paper’s charts. Qwen 2 0.5B runs at roughly 46 tokens per second on the Orin’s CPU cores and roughly 33 on the Pi, while Phi 3.5 mini at 3.8B parameters sits at about 5 on the Pi, about 10 on the Orin CPU and about 15 on the Orin GPU. The authors also contradict Meta’s claim that Llama 3.2 is optimised for Arm processors, since the advantage they saw for Llama 3.2 1B showed up in GPU inference and not on either CPU. Memory is where the measurements turn hardest. Peak usage grows close to linearly with model size, reaching about 7.9 GB for Gemma 2 9B on the Pi, and it flattens out on the Orin above Phi 3.5 mini because under 7 GB of the nominal 8 GB is usable once the operating system and background processes take their more than 1 GB. Raising the context window to 1028 tokens pushed the largest model into constant page faults on the CPU and made the Orin crash and restart during GPU inference. First-run load times ran from 5 seconds to over 200 seconds, and for Yi 1.5 6B and up the faster 4-bit format raised load times by 17 to 53 times, because those models get evicted from memory and re-read from the microSD card for every inference. Scaled to the 18.84 Wh battery of a Samsung Galaxy S24 Ultra at full utilisation, the March version of the paper puts Gemma 2 9B at 193 inference runs on the Pi and 101 on the Orin GPU, against 3,928 and 4,280 runs for Qwen 2 0.5B, with no configuration lasting 7 hours. The setup was built to keep thermal throttling out of the results rather than to observe it, so the authors fitted the Pi with an add-on heatsink after it overheated in early experiments, ran the fans on both boards at full speed and left 30 seconds between runs. Quantisation cost measurable quality. Perplexity, which tracks how uncertain a model is about its next token, rose against the 16-bit weights for every model and both 4-bit formats and rose most sharply for the smallest models, while the downstream benchmarks showed almost no accuracy drop. The paper is on arXiv and under review at an ACM journal. Update, November 20, 2025. The authors revised the paper after validating their power measurements against external hardware, which raised the measured consumption. The battery estimate for Gemma 2 9B falls from 193 runs to 75 on the Pi and from 101 to 73 on the Orin GPU, and no configuration now reaches 3.5 hours at full utilisation. The speed, memory and quality results are unchanged. Source: https://arxiv.org/abs/2503.09114 Read the article: https://llmobile.news/ticker/on-device-inference-tradeoffs/ResearchBenchmarksllama.cppQuantisationNvidiahttps://llmobile.news/ticker/pareto-q/https://llmobile.news/ticker/pareto-q/Tue, 04 Feb 2025 19:59:00 +0100Meta published ParetoQ on February 4, 2025, a training framework that puts 1-bit, 1.58-bit, 2-bit, 3-bit and 4-bit quantisation through the same recipe so the bit widths can be compared against each other rather than against methods built for one width. The result the authors draw from it is that 1.58-bit, 2-bit and 3-bit land close together and generally beat 4-bit on accuracy against model size, while 1-bit drops away. On LLaMA-3 8B the paper’s table gives an average of 71.2 across five commonsense tasks at 2 bits and 69.0 at 1.58 bits against 74.6 for the full-precision model, then 64.1 at 1 bit. That ranking only holds when size is counted in bytes rather than parameters, which is how a phone’s memory budget works. The authors measure each model by its effective quantised size, every weight counted at its own bit width plus the embeddings at theirs, and compare across that axis. Their example is a 2-bit MobileLLM-1B scoring 61.0 on an eight-task average against 59.2 for a 4-bit MobileLLM-600M, 1.8 points higher at a smaller effective size, and the paper reports the same pattern across LLaMA models up to 8B. The paper also describes where the training behaviour changes. At 3 bits and above the fine-tuned weights stay within 10 to 20 percent of the full-precision weights they started from and recover most of the accuracy after about 10B tokens, while at 2 bits and below they move by roughly 40 percent and need around 30B tokens, which the authors read as the network rebuilding its representations instead of nudging them. On that basis they argue against training ternary models from scratch, the route taken by BitNet b1.58 and Spectra, and put their ternary 600M model at 58.7 on a six-task average, against 57.5 for the ternary 3B model of the 1-bit Era and 58.2 for Spectra’s ternary 3.9B, both computed from the same table. For speed the authors wrote a 2-bit CPU kernel and timed five MobileLLM models on an Apple M1 MacBook Pro with 32 GB of memory using 6 threads, 5 prompt tokens and 122 generated tokens, and report the 2-bit curve sitting above the 4-bit one on accuracy against decoding speed without naming a factor. The one speedup they put a number on is on an H100 NVL GPU, where their 2-bit kernel built on CUTLASS runs 4.14 times faster than FP16 and 1.24 times faster than the 4-bit Machete kernel from vLLM at a 16384 by 16384 weight shape. At 1B, 3B and 8B the authors say the in-kernel conversion overhead cancels that gain and 4-bit gives the better speed-accuracy trade-off. Hardware is also why the paper favours 2-bit over ternary despite the two sitting together on its accuracy-size front. Storing three values needs either sparsity above 90 percent to pay for the indexing or packing that complicates the matrix multiply, and the authors note that some implementations store ternary weights as 2-bit signed integers anyway, which gives up the storage saving. They write that broad adoption will need “community-wide efforts, such as INT2 support in NVIDIA tensor cores”, and the paper itself names no weight release. Update, May 30, 2025. Meta published the ParetoQ checkpoints on Hugging Face, five MobileLLM sizes from 125M to 1.5B, each as a BF16 model plus 1-bit, 1.58-bit, 2-bit, 3-bit and 4-bit versions, 30 repositories in all. They sit behind a gated form under Meta’s FAIR Noncommercial Research License, which permits research use only. The training code had gone up on GitHub on March 3, 2025. Source: https://arxiv.org/abs/2502.02631 Read the article: https://llmobile.news/ticker/pareto-q/MetaResearchQuantisationMemoryBenchmarkshttps://llmobile.news/ticker/slms-can-still-pack-a-punch/https://llmobile.news/ticker/slms-can-still-pack-a-punch/Fri, 03 Jan 2025 20:53:57 +0100Shreyas Subramanian, Vikram Elango and Mecit Gungor of Amazon surveyed about 160 papers on small language models on January 3, 2025, covering general-purpose models under 8B parameters plus a few exceptions up to 13B. Their central claim is that some of these models reach the performance of models 10 to 100 times their parameter count, a ratio the authors call effective size and anchor to PaLM at 540B parameters. In their table, Phi-1 at 1.3B parameters is credited with an effective size of 175B on code benchmarks and Phi-2 at 2.7B with 70B. The survey is clear about where that stops. Its appendix prints completions from 14 models for the prompt “The diagonals of a rectangle”, where a 33M TinyStories model answers “and they were so beautiful” and a 125M GPT Neo model loops on its own words, while TinyLlama at 1.1B and Phi 1.5 at 1.3B give the correct definition. The authors conclude that several billion parameters are the minimum for usable general-purpose responses, and they caution that every effective size they report comes from each paper’s best reported benchmark, so results on other benchmarks can be much lower. The clearest wins they list are task-specific and stay inside one domain, with WizardMath at 7B beating Llama 2 70B on maths benchmarks and Code Llama at 7B beating it on coding benchmarks. On what closes the gap, the survey puts training data first. It credits Phi-1 with 50.6 percent on HumanEval at 1.3B parameters trained on textbook-quality synthetic data, and TinyGSM, where a 1.3B generator paired with a 1.3B verifier scored 81.5 on the GSM8K maths set against 77.4 for the GPT-3.5 model that wrote its training data. Distillation covers the training side, with Orca 2 at 13B leading Llama 2 70B by an average of 3.23 points after learning step-by-step explanations from a teacher model rather than just its answers. After training, the survey groups quantisation and pruning, citing SmoothQuant for 8-bit weights and activations at up to 1.56 times the speed and half the memory, and Sheared-LLaMA for cutting Llama 2 7B down to 1.3B and 2.7B using 3 percent of the compute that training those sizes from scratch would need. The authors name resource-constrained deployment as the main motivation for the whole field, listing mobile devices, edge computing and regions with limited connectivity, and point at SmolLM at 135M, 360M and 1.7B parameters as a family built to run locally on phones and laptops. They note that extreme quantisation and sparse training already put Llama inference on CPUs within reach, and that 8-bit Adam training of a 7B model fits on a single 24 GB Nvidia RTX 4090. The paper is on arXiv under a CC BY-NC-ND 4.0 license, and its authors maintain a leaderboard of small models on Hugging Face Spaces. Source: https://arxiv.org/abs/2501.05465 Read the article: https://llmobile.news/ticker/slms-can-still-pack-a-punch/ResearchBenchmarksDistillationQuantisationPhihttps://llmobile.news/ticker/exolabs-benchmarks/https://llmobile.news/ticker/exolabs-benchmarks/Wed, 25 Dec 2024 19:55:00 +0100EXO Labs opened a public benchmark site on December 25, 2024, as the first entry in a twelve-part series, publishing generation speed and time to first token measured on real consumer devices. EXO states these are measurements rather than estimates. In the data behind the site, an iPhone 15 Pro with 8 GB of RAM generates 14.0 tok/s on Llama 3.1 8B and takes 1.67 seconds to return its first token. A Galaxy S24 Ultra with 12 GB manages 5.6 tok/s on the same 8B model, with 4.16 seconds to the first token, according to EXO’s published results. On the smaller Llama 3.2 1B the iPhone reaches 56.1 tok/s and the Galaxy 22.5 tok/s. A single Mac mini M4 Pro with 24 GB sits at 49.3 tok/s on Llama 3.1 8B, which puts the phone at under a third of the Mac’s rate on the same model. EXO splits a model into contiguous slices of its layers and hands each slice to a different device, so a model too large for one machine can run across several. The activations passed from one device to the next are small, under 4 KB for Llama 3.2 3B, and EXO says the bottleneck is network latency rather than bandwidth. Pooling phones does not speed up a single reply. A cluster of two iPhone 15 Pros and one Galaxy S24 Ultra runs Llama 3.1 8B at 7.0 tok/s, below the 14.0 tok/s of one iPhone alone, which EXO attributes to network overhead when the model already fits on a single device. Three Mac mini M4 Pros run Llama 3.3 70B at 4.1 tok/s, a model that does not fit in the 24 GB of any one of them. Every entry carries a timestamp, a git commit and the identifier of the automated run that produced it, and the site is free to browse. EXO says it is expanding the suite towards mixed device types and different quantisations, and the exo runtime itself is on GitHub under Apache 2.0. Source: https://blog.exolabs.net/day-1/ Read the article: https://llmobile.news/ticker/exolabs-benchmarks/BenchmarksiPhoneAndroidLlamaApple Siliconhttps://llmobile.news/ticker/bluelm-v/https://llmobile.news/ticker/bluelm-v/Sat, 16 Nov 2024 01:14:51 +0100Researchers at CUHK MMLab and vivo AI Lab published BlueLM-V-3B on November 16, 2024, a vision-language model designed together with the code that runs it on a phone. The authors deployed it on the NPU of a MediaTek Dimensity 9300 in a vivo X100 and report a peak memory requirement of 2.2 GB and a generation speed of 24.4 tok/s with the language model weights quantised to 4-bit. The model pairs a 2.7B in-house BlueLM language model with a 400M SigLIP vision encoder that takes 384 by 384 image tiles. The authors report an average score of 66.1 on OpenCompass, a suite of eight multimodal benchmarks, which they state is the highest among models of 4B parameters or fewer, ahead of the 8B models MiniCPM-V-2.6 and InternVL2-8B. Most of the speed comes from how the model decides what resolution to feed the encoder. Dynamic resolution schemes pick a grid of tiles to fit an image, and the authors show that existing choices enlarge pictures a lot, with LLaVA-NeXT resizing and padding a 394 by 390 image to 768 by 768, four times the area, and InternVL 1.5 stretching a 380 by 76 image to 1920 by 384, twenty-five times the area. More pixels mean more tiles and more image tokens, so BlueLM-V-3B uses what the authors call relaxed aspect ratio matching, which tolerates a slightly smaller effective resolution when it wastes less of the canvas. Both examples then land on a single 1:1 tile, and across the 665k-sample LLaVA training set the method picks a smaller grid than InternVL 1.5 in 523k cases. The rest is shaped around what an NPU is bad at. The authors encode image tiles in batches of four at a time, which they measured as the fastest setting, and run the SigLIP convolution layer on the CPU in parallel with the transformer blocks on the NPU to hide 200 ms of latency. A downsampler merges every 2 by 2 patch into one token, cutting 729 tokens per tile to 196, and the prefill runs in chunks of 128 tokens rather than all at once. Precision is mixed, with INT4 for the language model weights, INT8 for the vision encoder and projector weights and for the KV cache, and INT16 and FP16 activations, which the authors say keeps accuracy stable because activations are more sensitive to quantisation than weights. Against the 8B MiniCPM-V 2.5 running on the CPU of the same chip through llama.cpp, the authors report 2.53 s to load and encode a 768 by 1536 image and 2.7 s of prefill, versus 4.0 s and 13.9 s, at 24.4 tok/s against 4.9 tok/s. The paper describes no public release of weights or code, and the authors say future work will target a broader range of mobile devices. Source: https://arxiv.org/abs/2411.10640 Read the article: https://llmobile.news/ticker/bluelm-v/MediaTekNPUQuantisationResearchBenchmarkshttps://llmobile.news/ticker/smollm2/https://llmobile.news/ticker/smollm2/Thu, 31 Oct 2024 19:30:00 +0100Hugging Face released SmolLM2 on October 31, 2024, a second generation of its small language models at 135M, 360M and 1.7B parameters. The company trained the 1.7B model on 11 trillion tokens, against 1T for the first SmolLM at the same size, and its model cards put the 360M model at 4T tokens and the 135M at 2T. The mixture combines FineWeb-Edu, DCLM and The Stack with datasets Hugging Face curated for this run, which its paper from February 2025 names as FineMath for mathematics, Stack-Edu for code and SmolTalk for instruction tuning. That paper states the 1.7B run used a multi-stage schedule that held the highest quality maths and code data back for the final tenth of training, from 10T to 11T tokens, while the 360M and 135M models trained in a single stage on a mix kept consistent throughout. Hugging Face fine-tuned the instruct versions on SmolTalk and then tuned them further on ranked answer pairs from UltraFeedback. Hugging Face reports the 1.7B base model at 60.5 on ARC against 49.2 for Llama 3.2 1B and 58.5 for Qwen2.5 1.5B, 36.7 on TriviaQA against 28.1 and 20.9, and 19.4 on MMLU-Pro against 11.7 and 13.7, all run zero-shot with its own lighteval harness and printed in the model card. The instruct version scores 56.7 on the IFEval instruction-following test against 53.5 for Llama 3.2 1B Instruct and 47.4 for Qwen2.5 1.5B Instruct, while Qwen2.5 stays ahead on the MT-Bench chat test at 6.52 to 6.13. At the smaller sizes Hugging Face compares the 360M model with Qwen2.5 0.5B, reporting 53.0 against 45.4 on ARC and 16.9 against 4.3 on TriviaQA, and publishes no third-party comparison for the 135M model. Table: Hugging Face's own figures for the base models. The weight files run 269 MB for the 135M model, 724 MB for the 360M and 3.42 GB for the 1.7B in bfloat16, and Hugging Face uploaded 4-bit ONNX builds the same day at 118 MB, 273 MB and 1.11 GB, plus a 4-bit GGUF of the 1.7B instruct model at 1.06 GB. The repository says the models run locally through llama.cpp, MLX, MLC and transformers.js. All six checkpoints carry an 8k context window and the Apache 2.0 license. Update, November 27, 2024. Hugging Face published a browser chat demo that runs the 1.7B instruct model through transformers.js and ONNX Runtime Web on WebGPU, with no server call after the download. Its local inference guide adds runnable code for llama.cpp, MLX and MLC, and lists the models on Ollama and in the PocketPal-AI phone app. Source: https://github.com/huggingface/smollm Read the article: https://llmobile.news/ticker/smollm2/Open weightsBenchmarksOpen sourcellama.cppWebGPUhttps://llmobile.news/ticker/small-model-training-bottlenecks/https://llmobile.news/ticker/small-model-training-bottlenecks/Fri, 25 Oct 2024 12:30:00 +0200Seven Apple researchers measured what it costs to train language models up to 2B parameters on rented Nvidia GPUs and found the most expensive card was the worst buy. The H100-80GB returned the fewest tokens per dollar of the three cards tested at every one of their four model sizes, 100M, 500M, 1B and 2B, while costing 2.53 times as much per hour as the A100-40GB. The price scale comes from normalising published hourly rates at Lambda Labs and Google Cloud, with the A100-40GB set at 1.00 and the A100-80GB landing at 1.46. The same comparison shows cost efficiency falling as GPUs are added, with a single card the best or near-best point at all four model sizes. For the 1B model, tokens per dollar drop from about 17,000 on one GPU to about 10,300 on 64. FlashAttention mattered more at small scale than at large. Reading the authors’ first figure, the 100M model gained roughly three times the tokens per dollar over vanilla attention, while at 2B the gain was under two times. The authors put this down to the cost of attention being quadratic in context length, so it dominates once the hidden dimension shrinks and the small models become bound by data movement between CPU and GPU and between GPUs. Vanilla attention also ran out of memory at a global batch size of 1024 on the 1B and 2B models, where FlashAttention did not. Two habits carried over from large-model training did not pay off. Plain Distributed Data Parallel, which keeps a full copy of the model on every GPU, beat both sharded FSDP variants for the smaller models because it moves less data, and FSDP only pulled ahead at 2B, where it kept training at a global batch size that made DDP run out of memory. Filling the cards was not cost-optimal either, since the batch size figure peaks around a global batch size of 128 for the 100M model, 32 for 500M, 16 for 1B and 8 for 2B, in every case well short of what the memory allowed. These are training runs on datacentre GPUs, not anything running on a phone, and the authors frame the work as guidance for labs with small budgets rather than for deployment. They name their own limits, running each setup at least three times for only 10 training steps, reporting normalised price ratios rather than absolute dollars, using no gradient accumulation, and leaving GPU utilisation and memory bandwidth measurements to future work. They also assume that settings such as the learning rate can be retuned to converge at whichever hardware configuration turns out cheapest. The 8-page paper is free to read on arXiv under the site’s non-exclusive distribution licence. Source: https://arxiv.org/abs/2410.19456 Read the article: https://llmobile.news/ticker/small-model-training-bottlenecks/AppleNvidiaResearchBenchmarksPyTorchhttps://llmobile.news/ticker/ministral/https://llmobile.news/ticker/ministral/Wed, 16 Oct 2024 10:00:00 +0200Mistral AI introduced Ministral 3B and Ministral 8B on October 16, 2024, two models the company built for on-device computing and at-the-edge use and announced together as les Ministraux. Mistral says Ministral 3B already outperforms its own Mistral 7B on most benchmarks, and reports the base model scoring 60.9 on MMLU against 52.4 for Gemma 2 2B and 56.2 for Llama 3.2 3B. Both models handle up to 128k tokens of context, which the announcement says was capped at 32k when run on vLLM. Ministral 8B uses what Mistral calls an interleaved sliding-window attention pattern, where a layer attends only to a limited span of recent tokens instead of the whole sequence, which the company says makes inference faster and lighter on memory. Mistral reports the instruction-tuned Ministral 3B at 8.1 on MTBench, 64.3 on Arena Hard and 77.4 on HumanEval, the last of those above Gemma 2 2B at 42.7 and Llama 3.2 3B at 61.0. Ministral 8B reaches 8.3 on MTBench and 70.9 on Arena Hard in the same table, while Gemma 2 9B stays ahead on WildBench at 43.8 against 41.3. All of these are Mistral’s own numbers, produced with an internal framework the company says it used to re-evaluate every model for the comparison. Table: Mistral AI's own figures, from its internal evaluation framework. Mistral names on-device translation, smart assistants that work without an internet connection, local analytics and autonomous robotics as the applications customers and partners asked for, citing demand for local, privacy-first inference. The company also positions the two models as intermediaries in front of larger ones such as Mistral Large, tuned to parse input, route tasks and call APIs across multiple contexts in multi-step agent workflows. Both models went live through Mistral’s API on the day of the announcement, at $0.04 per million tokens for Ministral 3B and $0.1 per million for Ministral 8B, with input and output priced the same. Mistral published weights for Ministral 8B Instruct only, for research use, and lists that model under both the Mistral Research License and the Mistral Commercial License, while Ministral 3B appears under the commercial license alone with no weights release. Anyone wanting to self-deploy either model has to contact Mistral for a commercial license, and the company says it will help with lossless quantisation for specific use cases. Source: https://mistral.ai/news/ministraux/ Read the article: https://llmobile.news/ticker/ministral/MistralBenchmarksTranslationAgentshttps://llmobile.news/ticker/palmbench/https://llmobile.news/ticker/palmbench/Sat, 05 Oct 2024 05:37:00 +0200Researchers at the University of Wisconsin-Madison, Amazon Web Services AI and Uber ran quantised language models across eight phones and edge boards and measured the widest gap between Apple and everything else. In PalmBench, published as a conference paper at ICLR 2025, TinyLlama-1.1B at 4-bit reached close to 100 tok/s on an iPhone 15 Pro against roughly 30 tok/s on the fastest Android phone in the set, a Pixel 7. The authors report that the three-year-old iPhone 12 Pro also stayed ahead of every Android device and of Nvidia’s Jetson Orin Nano. Chart: Li et al., licensed CC BY 4.0. Throughput measured with MLC LLM, which was the only framework that ran on all eight platforms. The test bed covers a Pixel 4, Pixel 5a, Pixel 7, Galaxy S22 Ultra, iPhone 12 Pro, iPhone 15 Pro, an Orange Pi 5 and a Jetson Orin Nano, each with 6 GB or 8 GB of memory. Models include Llama-2-7B, Llama-3 and 3.1-8B, Llama-3.2-3B, Mistral-7B, Vicuna, Qwen2, Phi-2, Gemma-2-2B, RedPajama-INCITE-3B and TinyLlama-1.1B, quantised from 2-bit to 8-bit and additionally with GPTQ, AWQ and FasterTransformer. The team used two runtimes, MLC LLM and llama.cpp, and states that llama.cpp does not run on iPhone, so the cross-platform numbers come from MLC. Quantisation costs more answer quality than the bit count alone suggests. Scoring each quantised build’s exact match against the unquantised model’s own answers on SQuAD and Natural Questions, the paper’s figure puts 4-bit builds between about 50 and 61 percent, 6-bit between roughly 62 and 78 percent, and 2-bit and 3-bit builds as low as 15 percent. The authors note that 3-bit did not clearly beat 2-bit, and that 5-bit landed close to 4-bit while consuming more resources and degrading more than any 4-bit variant. Chart: Li et al., licensed CC BY 4.0. Exact match is measured against the answers of the same model before quantisation, not against ground truth. The 4-bit builds drew more power than the 3-bit ones rather than less. The paper’s table records 4-bit consuming more energy on all eight devices for the same 3B Llama model, rising from 10.13 to 13.05 mWh on the iPhone 15 Pro and from 14.54 to 14.68 mWh on the Pixel 7, which the authors put down to the higher-precision build occupying more GPU duty cycles. Surface temperature measured with a FLIR C5 thermal camera peaked at 75.4 °C on the Orange Pi 5 and 46.3 °C on the iPhone 15 Pro at 4-bit. Harmful output tracked the bit count more cleanly, with the hallucination rate on Llama-3-8B at 34.7 percent for 2-bit against 9.1 percent for 4-bit GPTQ and 7.9 percent at 8-bit. The authors name several limits on their own work. They cannot compare the two runtimes on iOS because llama.cpp does not support the iPhone, they profiled total memory only at 3-bit and 4-bit because higher-bit MLC builds exceeded the devices’ capacity, and they dropped models that would not fit at all, such as the unquantised Llama-2-7B at 13.11 GB. They also note that both runtimes expose few decoding options, with beam search unavailable in llama.cpp. Artificial Analysis has since run an independent benchmark of 33 quantised models on a single iPhone 17 Pro. Source: https://arxiv.org/abs/2410.05315 Read the article: https://llmobile.news/ticker/palmbench/BenchmarksQuantisationiPhoneAndroidMLC LLMhttps://llmobile.news/ticker/gemma-2/https://llmobile.news/ticker/gemma-2/Wed, 31 Jul 2024 17:00:00 +0200Google released Gemma 2 2B on July 31, 2024, an open model of 2.6B parameters that Google says produces outsized results by learning from larger models through distillation. Google reports it at 1126 Elo on the LMSYS Chatbot Arena leaderboard, captured on July 30, 2024, ahead of Mixtral 8x7B Instruct at 1114, GPT 3.5 Turbo 0314 at 1106 and Llama 2 70B chat at 1093, and states that the model surpasses all GPT-3.5 models there. Chart: Google, from the announcement. Scores are Google's own reading of the leaderboard on July 30, 2024. The technical report describes training the 2B size on 2 trillion tokens with knowledge distillation, where a bigger teacher model supplies a probability for every token in context and the small student is trained to match those probabilities rather than to predict the next token. In an ablation the authors ran, a 2B model trained over 500B tokens distilled from a 7B teacher averaged 67.7 across three benchmarks, against 60.3 for the same model trained from scratch. The released model holds 2.02B non-embedding parameters plus 590M embedding parameters, which come from the 256k-entry Gemini vocabulary, and it interleaves local sliding-window attention with global attention over an 8192-token context. Google DeepMind trained it on 512 TPUv5e chips. Google names edge devices and laptops as deployment targets next to cloud serving on Vertex AI and Google Kubernetes Engine, and says the model is small enough for the free tier of T4 GPUs in Google Colab. For running it locally Google points to Ollama and to Gemma.cpp, its own lightweight C++ inference engine, and says support in MediaPipe, the on-device inference stack for Android, iOS and the web, is coming. Google adds that NVIDIA optimised the model with the TensorRT-LLM library for GeForce RTX cards and for Jetson modules at the edge. Google put the weights on Kaggle, Hugging Face and Vertex AI Model Garden under the Gemma terms, which it calls commercially friendly for research and commercial applications, and made the model available to try in Google AI Studio. The same announcement added ShieldGemma, a set of safety classifiers built on Gemma 2 that filter hate speech, harassment, sexually explicit and dangerous content, and Gemma Scope, over 400 sparse autoencoders for inspecting the inner workings of Gemma 2 2B and 9B. Source: https://developers.googleblog.com/en/smaller-safer-more-transparent-advancing-responsible-ai-with-gemma/ Read the article: https://llmobile.news/ticker/gemma-2/GoogleGemmaDistillationOpen weightsBenchmarkshttps://llmobile.news/ticker/spectra/https://llmobile.news/ticker/spectra/Wed, 17 Jul 2024 07:53:00 +0200Nolano AI, the University of Montreal and Mila published Spectra on July 17, 2024, a suite of 54 language models covering nine parameter counts from 99M to 3.9B, each trained on the same 300B tokens of SlimPajama. Every size exists as a half-precision model the authors call FloatLM, as GPTQ-quantised copies at 3, 4, 6 and 8 bits called QuantLMs, and as a TriLM, a model trained from scratch with every linear-layer weight restricted to -1, 0 or 1. The paper’s size table puts TriLM 3.9B at 10.76 billion bits, roughly 1.3 GB, against 63.83 billion bits for the half-precision model of the same parameter count, a factor of 5.9. That is the comparison the suite is built for, since what fits on a device is bits rather than parameters. Averaged over six commonsense and reasoning benchmarks, the paper’s tables put TriLM 3.9B at 60.7 against 61.4 for FloatLM 3.9B, level with the 4-bit QuantLM 3.9B at 60.7 which needs 20.59 billion bits. Set against the models nearest to it in size, TriLM 3.9B beats the half-precision 830M model, 60.7 against 53.3 at 13.34 billion bits, and the 3-bit QuantLM 2.4B, 60.7 against 54.7 at 10.95 billion bits. On LAMBADA it reaches 61.6 against 61.1 for FloatLM 3.9B, and on TriviaQA 21.3 against 21.5. Chart: Kaushal et al. Averages across Arc Easy, Arc Challenge, BoolQ, HellaSwag, PIQA and WinoGrande. During training a TriLM keeps floating-point latent weights and ternarises them on the fly in the forward pass, with one scale value per weight matrix shard and gradients passed through by a straight-through estimator, so only the ternary states and the scales are needed at inference. Embeddings and the output head stay in half precision across the suite. The authors position TriLM against BitNet b1.58, which also trains ternary weights from scratch, and say they normalise once before each linear layer instead of normalising, scaling and quantising activations to 8 bits four to seven times per transformer block; their own replication of the BitNet architecture at 1.1B, trained on 100B tokens, scores below TriLM 1.1B on the same benchmarks. The paper names the weaknesses it measured. At 1.1B parameters the ternary model does not reach its half-precision counterpart, 51.6 against 54.9 on the six-benchmark average, and the 2.4B one still trails at 57.3 against 59.2, so the match with a half-precision model of the same parameter count holds at 3.9B only. On CrowsPairs, which measures how often a model prefers a stereotyped sentence, TriLM 3.9B lands at 65.4 percent against 64.7 for FloatLM 3.9B and 62.6 for the smaller FloatLM 830M, so the ternary model stereotypes more than the half-precision model closest to it in bits. No model in the suite beats random guessing on TruthfulQA, and while TriLM 3.9B holds up on Penn Treebank and LAMBADA perplexity, it stays behind FloatLM 3.9B on web corpora including Dolma, C4 and RefinedWeb, which the authors read as weaker memorisation of training data. The weights and more than 500 intermediate checkpoints are on Hugging Face under Apache 2.0. The TriLM checkpoints are published unpacked to FP16, which the model cards say makes them compatible with ordinary half-precision matrix multiplication and leaves the released files at full half-precision size. The authors list smartphones, laptops and automobiles among the resource-constrained devices they see for ternary models, and end by calling for work on the toxicity, stereotyping and web-corpus gaps they measured. Source: https://arxiv.org/abs/2407.12327 Read the article: https://llmobile.news/ticker/spectra/ResearchQuantisationMemoryOpen weightsBenchmarkshttps://llmobile.news/ticker/smollm/https://llmobile.news/ticker/smollm/Tue, 16 Jul 2024 16:00:00 +0200Hugging Face released SmolLM on July 16, 2024, a family of three base models at 135M, 360M and 1.7B parameters. The team says the models are designed to be small enough to run locally across a range of hardware, from smartphones to laptops, and cites the iPhone 15 with 6 GB of DRAM and the iPhone 15 Pro with 8 GB as reference points. Hugging Face trained the 135M and 360M models on 600B tokens each and the 1.7B model on 1T tokens. The substance is the training data, released as SmolLM-Corpus. Cosmopedia v2 contributes 28B tokens of synthetic textbooks, stories and articles generated by Mixtral-8x7B-Instruct, 39 million documents built from a predefined list of 34,000 topics taken from the BISAC book classification instead of from unsupervised clustering. FineWeb-Edu contributes 220B deduplicated tokens of educational web pages, pulled out of FineWeb by a quality classifier trained on annotations from Llama3-70B-Instruct. Python-Edu applies the same idea to code, keeping 4B tokens of the 40B Python tokens in The Stack that scored 4 or higher on an educational code classifier. On Hugging Face’s own evaluation, run with the same lighteval setup for every model it could download, SmolLM-135M averages 39.50 across MMLU, TriviaQA, ARC, PIQA, HellaSwag, OpenBookQA and Winogrande, against 32.75 for GPT2-137M and 32.49 for Pythia-160M. SmolLM-360M averages 44.44 against 39.60 for Qwen2-500M and 39.32 for Qwen1.5-500M, and SmolLM-1.7B averages 51.97 against 50.36 for Microsoft’s Phi-1.5 and 47.98 for Qwen2-1.5B. Meta’s MobileLLM is not publicly available, so Hugging Face took its per-benchmark numbers from the paper and reports no average for those rows. Table: Hugging Face's own figures, evaluated with the same setup for every model except MobileLLM. Hugging Face also benchmarked the memory footprint of each size. SmolLM-135M takes 269.03 MB in bf16, 162.87 MB at int8 and 109.78 MB at int4. SmolLM-360M takes 723.65 MB, 409.07 MB and 251.79 MB across the same three precisions, and SmolLM-1.7B takes 3422.76 MB, 1812.14 MB and 1006.84 MB. Table: Hugging Face. The weights are on Hugging Face under Apache-2.0, with ONNX checkpoints published beside the transformers ones and a GGUF build for llama.cpp announced as planned. Hugging Face also put up browser demos of the instruction-tuned 135M and 360M models that run client-side through WebGPU, with no server call. The corpus is released separately under ODC-By. Source: https://huggingface.co/blog/smollm Read the article: https://llmobile.news/ticker/smollm/Hugging FaceOpen weightsBenchmarksMemoryWebGPUhttps://llmobile.news/ticker/mobileclip/https://llmobile.news/ticker/mobileclip/Fri, 21 Jun 2024 02:15:00 +0200Five Apple researchers presented MobileCLIP at CVPR 2024 on June 20, a family of image-text models built for phones. They exported each model with Core ML Tools 7.0 and timed it at batch size 1 on an iPhone 12 Pro Max running iOS 17.0.3, and their Table 7 gives the smallest variant, MobileCLIP-S0, 1.5 ms for the image encoder and 1.6 ms for the text encoder at 67.8% zero-shot accuracy on ImageNet. The same authors measure OpenAI’s ViT-B/16 CLIP on that phone at 11.5 ms and 3.3 ms for 68.3%, and both models average 58.1 across the 38 datasets of the DataComp evaluation suite. Multi-modal reinforced training, the method the paper introduces, moves the expensive part of distillation out of the training loop and into the dataset. Apple reinforces the DataComp dataset once, writing five synthetic captions per image with the CoCa model in OpenCLIP, storing the parameters of strong random image augmentations, 30 per image for the 12M subset and 10 for the billion-sample set, and storing the embeddings that two ViT-L/14 teacher models produce for those augmented images and for the real and synthetic captions. Training afterwards reads the captions and the teacher embeddings from disk, so the captioning model and the teachers never run again however many models are trained on the set. The paper puts numbers on what that saves. One epoch over the 12.8M samples of the reinforced set on a single node of 8 A100-80GB GPUs takes 1.3 hours, the same as an epoch on the unreinforced data, against 21.1 hours when the captioning model and the teacher ensemble run during training, according to Table 4d. The cost moves to storage, and Table 4c puts DataComp-1B at 90 TB against 140 TB for the reinforced DataCompDR-1B, with the 12M subset going from 0.9 TB to 1.9 TB. MobileCLIP comes in four sizes, and Table 7 lists image and text encoder parameters separately. S0 pairs an 11.4M-parameter image encoder with a 42.4M-parameter text encoder, S1 uses 21.5M and 63.4M at 2.5 ms and 3.3 ms, S2 uses 35.7M and 63.4M at 3.6 ms and 3.3 ms, and MobileCLIP-B keeps a standard ViT-B/16 image encoder at 86.3M and 10.4 ms. Zero-shot ImageNet accuracy across the four runs 67.8%, 72.6%, 74.4% and 76.8%, and a version of B trained on 39B seen samples instead of 13B reaches 77.2%. All three small variants replace the vision transformer with a convolution-transformer hybrid derived from Apple’s FastViT, and S0 also swaps the text encoder for one that mixes 1-D convolutions with four self-attention layers, which the authors report as 5% smaller and 15.8% faster than the 12-layer transformer text encoder that S1 and S2 keep. Apple released the code, the weights and the reinforced data at apple/ml-mobileclip, with checkpoints for all four variants and the longer-trained B, and with DataCompDR-1B and DataCompDR-12M on Hugging Face as synthetic captions, augmentation parameters and teacher embeddings keyed to DataComp images rather than as the images themselves. The repository lists the code under MIT, the models under Apple’s own machine learning research model terms, and the data under CC BY-NC-ND. Apple showed zero-shot scene classification running in real time on an iPhone in the CVPR exhibit hall, and added an iOS app for the same task to the repository in November 2024. Source: https://openaccess.thecvf.com/content/CVPR2024/html/Vasu_MobileCLIP_Fast_Image-Text_Models_through_Multi-Modal_Reinforced_Training_CVPR_2024_paper.html Read the article: https://llmobile.news/ticker/mobileclip/AppleiPhoneResearchBenchmarksOpen weightshttps://llmobile.news/ticker/fox/https://llmobile.news/ticker/fox/Thu, 13 Jun 2024 15:00:00 +0200TensorOpera released Fox-1 on June 13, 2024, a decoder-only language model with 1.6B parameters trained on 3 trillion tokens of text and code. The company says the run followed a three-stage data curriculum at 8K sequence length, without saying what each of the three stages contained. Fox-1 uses grouped query attention, where several attention heads share one set of key and value vectors to cut memory traffic, with 16 attention heads over 4 key-value heads. TensorOpera describes the model as deeper than others of its size, 78% deeper than Gemma-2B, 33% deeper than Qwen1.5-1.8B and StableLM-2-1.6B, and 15% deeper than Apple’s OpenELM-1.1B. TensorOpera reports an average of 47.13% across ARC Challenge, HellaSwag, MMLU, TruthfulQA, Winogrande and GSM8k, which it ran in the Open LLM Leaderboard setup on one machine with 8 H100 GPUs. It puts Qwen1.5-1.8B at 46.81%, Gemma-2B at 46.36%, StableLM-2-1.6B at 45.92% and OpenELM-1.1B at 38.28%. Fox-1 leads that group on GSM8k with 36.39% against 34.04% for Qwen1.5-1.8B and 17.06% for Gemma-2B, and trails Qwen1.5-1.8B on MMLU with 43.05% against 47.15%. Chart: TensorOpera's own figures. The device claim in the announcement is a target rather than a measurement. TensorOpera puts the model’s FP16 memory use at 3,703 MiB, against 5,379 MiB for Gemma-2B, 4,739 MiB for Qwen1.5-1.8B and 3,852 MiB for StableLM-2-1.6B, and says that makes Fox-1 comparable for on-device deployment. The stated plan is that developers train on the company’s cloud platform and then deploy, personalise and monitor models on smartphones and AI-enabled PCs through its FedML platform. No figure for speed or latency on a phone appears in the post, and the throughput number it does give, over 200 output tokens per second per user, was measured with vLLM on a single H100. Chart: TensorOpera's own figures. TensorOpera published the base model under the Apache 2.0 license on Hugging Face and on its own platform, and said an instruction-tuned version would follow. Source: https://blog.tensoropera.ai/tensoropera-unveils-fox-foundation-model-a-pioneering-open-source-slm-leading-the-way-against-tech-giants/ Read the article: https://llmobile.news/ticker/fox/TensorOperaOpen weightsBenchmarksMemoryhttps://llmobile.news/ticker/llms-on-mobile-measurements/https://llmobile.news/ticker/llms-on-mobile-measurements/Tue, 11 Jun 2024 15:00:00 +0200Five researchers at Beijing University of Posts and Telecommunications measured language model inference on four Android phones and presented the results at the EdgeFM workshop of MobiSys 2024, deploying 22 models from 0.5B to 7B parameters through llama.cpp. For the 18 models they charted at 4-bit quantisation, accuracy for most of them sat between 55 and 60 percent, averaged across Winogrande, HellaSwag and MMLU, and latency came out at about 200 ms per token averaged over the four devices. The phones were a Xiaomi 14 with a Snapdragon 8 Gen 3 and 16 GB of RAM, a Pixel 7 Pro with a Tensor G2 and 12 GB, a Xiaomi 12S with a Snapdragon 8 Gen 1+ and 12 GB, and a Meizu 18 Pro with a Snapdragon 888 and 8 GB. Most of the 7B models needed about 4 GB of memory, which the authors put at half or a third of a current phone’s RAM. Their memory leaderboard has Bloom-7B highest, at 1.27 times the footprint of MPT-7B at the same parameter count, and Qwen1.5 0.5B lowest at under 1 GB. Memory use did not move with input or output length in their runs, with Baichuan holding at 4.83 GB whether the prompt was 10 or 50 tokens long and whether 1 or 1000 tokens were generated, which they trace to llama.cpp allocating one large pool at startup rather than sizing it per phase. Newer chips helped the two inference phases unevenly. Against the Snapdragon 888, the authors measured the Snapdragon 8 Gen 3 raising prefill throughput on LLaMA 2 by 81% but decode throughput by only 43%, with the Snapdragon 8 Gen 1+ at 45% and 34%. Decode is the phase that emits one token at a time and so governs how fast text appears on screen, and there the gap between the 8 Gen 3 and the 8 Gen 1+ was 7%. Continuous prefill runs also pushed the Xiaomi 12S, Xiaomi 14 and Meizu 18 Pro into thermal throttling, with per-token times drifting during a test series. The processor comparison went against the GPU in most cases. In the engine test llama.cpp on the 8 Gen 3 CPU prefilled 1.63 times faster than MLC LLM on the same phone’s GPU, 10.3 tokens per second against 6.3, while MLC on the GPU decoded faster at 9.8 against 6.6. On the mid-range Snapdragon 7 Gen 2 the CPU was ahead of the GPU in both phases, 4.59 and 4.65 against 3.3 and 3.5. The authors put this down to mobile GPUs being built for rendering rather than general-purpose compute, and to the extra cost of moving data. Quantisation cost less than the hardware choices did. Testing eight bit-widths on LLaMA 2, the authors report 4-bit as the best compromise, giving up 0.6% average accuracy against the 32-bit model while cutting model size 8 times, where 2-bit halves the size again for four times the accuracy loss. Code and results are published as TinyLLMLeaderBoard on GitHub, and the six-page paper is free to read in the ACM Digital Library. Source: https://dl.acm.org/doi/10.1145/3662006.3662059 Read the article: https://llmobile.news/ticker/llms-on-mobile-measurements/BenchmarksResearchllama.cppQualcommAndroidhttps://llmobile.news/ticker/mobile-foundation-model-as-firmware/https://llmobile.news/ticker/mobile-foundation-model-as-firmware/Wed, 29 May 2024 15:00:00 +0200Researchers at Beijing University of Posts and Telecommunications proposed that a phone ship one shared multimodal model instead of letting every app bundle its own, in a paper published in the ACM MobiCom 2024 proceedings on May 29, 2024. The operating system and the hardware co-manage that model like firmware, unchangeable by apps or by the OS itself, exposed to applications as a system service, and each app reaches it through a small adapter fine-tuned offline for its own task. Their prototype, called M4, holds 9.2B parameters and needs 7.5 GB of peak memory, and the authors report it reaching accuracy comparable to purpose-built models on 85% of the 50 datasets in a benchmark they assembled from 38 mobile AI tasks across five input types. What the shared model replaces is one small model per app per task. The paper’s baselines are 50 task-specific models of 1M to 500M parameters each, one per dataset, against which M4’s adapters run from 1,000 to 10 million parameters, so each added task costs under 10 MB. Measured on an Nvidia Jetson Orin NX, 4-bit M4 needs 6.1 GB of storage to serve all 50 tasks against 15.2 GB for the 50 separate models, with the crossover at about 15 tasks, and 7.5 GB of peak memory against roughly five times that. The authors state that on a device with 12 GB of memory the 4-bit model plus all 50 adapters fits, where only 20 of the 50 task-specific models would. The prototype is slower than the models it replaces. On the Jetson Orin NX with 16 GB, the authors measured M4 averaging 18 times the inference latency of the task-specific models across the 50 tasks and 19 times the energy, 3.6 s against 0.2 s. On a Pixel 7 Pro CPU they measured an average of 6.8 s against 0.54 s, and their per-task breakdown puts image classification at 2.10 s and question answering at 6.34 s to the first token and 0.24 s per token after it. They state that M4 cannot currently run on a stock smartphone GPU or NPU at all, because those processors lack support for the operators it uses. The NPU numbers in the paper are a projection rather than a measurement. The authors estimate that M4 on an NPU would average 0.48 s and 1.3 J, under the 0.54 s and 2.9 J they measured for task-specific models on the Pixel 7 Pro CPU, but they derive that by applying the CPU-to-NPU ratio they observed for task-specific models, not by running M4 on an NPU. Their case for a simpler accelerator rests on a separate Pixel 7 Pro measurement, where they converted 110 downloaded models to TensorFlow Lite and only 8% ran entirely on the NPU, those gaining a median speedup above 20 times over the CPU. M4 itself uses 39 operator types against the 156 that the 50 task-specific models need between them. The authors name their own limits. They write that the accuracy results come from an A100 and the Jetson board rather than from phones, that M4 underperforms task-specific models on some tasks including translation, and that a prototype assembled from off-the-shelf pre-trained models is “still highly inefficient in terms of accuracy and model parameter size”. Its backbone is Meta’s LLaMA-7B at 8-bit, with encoders taken from ImageBind and Whisper, and they note that adapters trained against one backbone stop working when the backbone is upgraded, so the design still needs a stable interface between the two. Code and benchmark are published at github.com/UbiquitousLearning/MobileFM, and the paper carries ACM copyright rather than an open license. Source: https://dl.acm.org/doi/10.1145/3636534.3649361 Read the article: https://llmobile.news/ticker/mobile-foundation-model-as-firmware/ResearchBenchmarksNPUPixelLlamahttps://llmobile.news/ticker/openelm/https://llmobile.news/ticker/openelm/Tue, 23 Apr 2024 01:12:00 +0200Apple posted OpenELM to arXiv on April 22, 2024, a family of four language models at 270M, 450M, 1.1B and 3B parameters, each also released in an instruction-tuned version. The name stands for Open-source Efficient Language Models, and the efficiency comes from layer-wise scaling, which spreads the parameter budget unevenly across the transformer’s layers instead of giving every layer the same shape. Table 1 of the paper gives the 1.1B model an average of 45.93% across the OpenLLM leaderboard tasks against 43.57% for OLMo 1.2B, a gap the authors state as 2.36%, reached on 1.5T pre-training tokens against OLMo’s 3.0T. Layer-wise scaling varies the number of attention heads and the width of the feed-forward network from layer to layer, keeping layers near the input narrower and widening them towards the output, so a layer late in the stack holds more parameters than an early one. Apple trained the four variants for 350k iterations with its CoreNet library on public data only, drawing on RefinedWeb, a deduplicated PILE, a subset of RedPajama and a subset of Dolma v1.6, about 1.8T tokens in total. Text is filtered and tokenised on the fly rather than pre-tokenised, using the same tokeniser as Llama. The paper benchmarks the models on an Apple MacBook Pro with an M2 Max chip and 64 GiB of RAM running macOS 14.4.1, through a port of OpenELM to MLX v0.10.0, and separately on a Linux workstation with an Intel i9-13900KF and an NVIDIA RTX 4090. On the MacBook in BFloat16, Apple measures the 270M model generating 212.40 tok/s and the 3B model 33.96 tok/s, with the 4-bit versions at 256.35 and 60.33 tok/s. On the CUDA machine OpenELM 1.08B is slower than OLMo 1.18B, at 92.15 tok/s against 203.40, which the authors attribute to a naive RMSNorm implementation and to OpenELM having 113 normalisation layers where OLMo has 33. Apple presents the release as the complete framework rather than weights and inference code, covering the training and evaluation code for public datasets, the training logs, multiple checkpoints, the pre-training configurations and the code that converts a model to MLX for inference and fine-tuning on Apple devices, with pre-converted bf16 and 4-bit checkpoints for all four sizes. The eight models are on Hugging Face and the code is in CoreNet. Both the model cards and CoreNet name the terms as the Apple sample code license, which is Apple’s own license and not one of the standard open source licenses. Source: https://arxiv.org/abs/2404.14619 Read the article: https://llmobile.news/ticker/openelm/AppleApple SiliconResearchOpen weightsBenchmarkshttps://llmobile.news/ticker/octopus-function-calling/https://llmobile.news/ticker/octopus-function-calling/Tue, 02 Apr 2024 03:29:28 +0200Wei Chen, Zhiyuan Li and Mingyuan Ma of Stanford University and Harvard University posted Octopus on April 2, 2024, four open models fine-tuned to pick the right software API for a request and fill in its arguments. On a benchmark the authors built themselves, they report their fine-tuned Gemma 2B answering 93 percent of calls correctly against 96 percent for GPT-4 and 50 percent for GPT-3.5, with the CodeLlama 7B and Gemma 7B versions reaching 97 percent once an output filter is switched on. The title says on-device, but nothing in the paper runs on a device. The authors fine-tuned the models on A100 80GB GPUs and queried GPT-4 and GPT-3.5 through OpenAI’s API, so every number was measured server-side. The device claim is a single sentence stating that models of this size can already be deployed on mobile, pointing at MLC LLM, and the paper names no phone, no chip and no latency, memory or throughput figure. The training set starts from roughly 30,000 of the most used APIs on RapidAPI Hub, which GPT-4 cut to about 20,000 by dropping entries with missing arguments or descriptions that did not match their parameters. The authors rewrote each API as a Python function signature with a docstring, had GPT-4 write five answerable queries per API and an equal share of unanswerable ones, and let GPT-4 verify the result, ending at about 150,000 examples as described in the paper. Vector search over the function descriptions pulled in near-identical functions as distractors, and training worked through them from easy to hard. Base models were CodeLlama 7B, Gemma 2B and 7B and Stable Code 3B, all tuned with LoRA at rank 16 and 8-bit quantisation, using 90 GPU hours for CodeLlama 7B and Gemma 7B, 60 for Stable Code 3B and 30 for Gemma 2B. The output filter the authors call a conditional mask blocks, at each generated token, anything that would break the expected format, so a function name has to come from the known list and an argument has to match its declared type. The authors state that it cannot be applied to GPT-4 or GPT-3.5 because the OpenAI API does not return the underlying token scores, and that GPT-3.5 was not counted wrong for formatting mistakes. The paper states that the training dataset and the fine-tuned models will be open sourced soon, and names no repository or date. The preprint went up under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 license. Source: https://arxiv.org/abs/2404.01549 Read the article: https://llmobile.news/ticker/octopus-function-calling/Nexa AIResearchAgentsGemmaBenchmarkshttps://llmobile.news/ticker/merino-iot-language-models/https://llmobile.news/ticker/merino-iot-language-models/Wed, 28 Feb 2024 04:20:00 +0100Researchers at the University of Central Florida and the University of Technology Sydney published MeRino on February 28, 2024, a method that draws up a transformer architecture to fit a given device’s limits on parameters, arithmetic and latency, instead of shrinking an existing model to fit. Their largest design, MeRino-64M, reaches the same 0.408 average accuracy across twelve zero-shot tasks as Meta’s OPT-350M while carrying 64M parameters against 331M. The authors time it at 114 ms against 559 ms for a 128-token input on an NVIDIA Jetson Nano, or 4.9 times faster. The search scores a candidate architecture without ever training it. Each candidate shape, meaning how many blocks it has and how wide and how deep each block is, gets a single number that the authors compute from the dimensions of its attention and feedforward matrices and treat as a measure of how much information the network can represent, with a second term that penalises shapes which are deep and narrow, because those train badly. An evolutionary algorithm mutates block widths and depths for 100,000 rounds, discards everything that breaks the budget, keeps the highest-scoring survivors and returns the best one. A precomputed lookup table stands in for the matrix decomposition the score would otherwise need, and the authors put its average error at 0.03 percent. Because the score is arithmetic over an architecture’s dimensions rather than a forward pass, the search runs on the target board itself. The paper puts one MeRino search at 0.05 hours and 0.75 Wh on the Jetson Nano’s CPU, against 1.2 hours and 300 Wh for the TE-NAS architecture search on an NVIDIA GTX 1080Ti. Every device measurement in the paper comes from that Jetson Nano developer board, taken at batch size 1 and sequence length 128 and averaged over 16 runs, and the authors report no measurements on phones or tablets. Training the designed models is separate and conventional work, at 600,000 steps on 8 NVIDIA H100 GPUs. The same tables show MeRino-64M behind OPT-350M on language modelling, at 22.47 against 18.51 perplexity on WikiText-2 and 27.06 against 23.08 on Penn Treebank, where lower is better. The smallest design, MeRino-52M, matches Pythia-70M at 0.383 average zero-shot accuracy and 48 ms against 95 ms. The authors state that the weights and evaluation for Meta’s MobileLLM were not released when they submitted, so they do not compare against it, and they report that 8-bit weight quantisation leaves accuracy almost unchanged while 4-bit costs about 1 percent. The work was accepted at AAAI 2025 and the paper names no code release. Source: https://arxiv.org/abs/2403.07921 Read the article: https://llmobile.news/ticker/merino-iot-language-models/ResearchBenchmarksNvidiaMemoryhttps://llmobile.news/ticker/stable-lm-2-zephyr/https://llmobile.news/ticker/stable-lm-2-zephyr/Tue, 27 Feb 2024 20:00:00 +0100Stability AI published the Stable LM 2 1.6B technical report on February 27, 2024, describing a decoder-only model of 1.6B parameters trained from scratch on roughly 2 trillion tokens of public data. The mix is multilingual by design, with web text in German, Spanish, French, Italian, Dutch and Portuguese alongside English, and the report states the finished model handles all seven. The model has 24 layers and a context length of 4096 tokens, which the authors list as one of the things they want to extend. In the report’s multilingual table, which averages translated versions of four English benchmarks and a next-word prediction set across those seven languages, the base model scores 40.5, above Google’s Gemma 2B at 39.8 with 2.5B parameters and Microsoft’s Phi-2 at 34.6 with 2.7B. The chat variant reaches 41.5 and Stability’s own 3B model 41.7. The authors state they used no multilingual data during fine-tuning, so the language coverage comes out of pre-training alone. On the six English tasks of the Hugging Face Open LLM Leaderboard, the same report’s table puts the base model at an average of 45.3, behind Alibaba’s Qwen 1.5 1.8B at 46.6 and Gemma 2B at 46.5. The instruction-tuned StableLM 2 Zephyr 1.6B averages 49.7, the highest of the sub-2B entries in that table, ahead of Phi-1.5 at 47.7 and behind the 2.7B Phi-2 at 61.3. The report’s abstract calls Stable LM 2 1.6B the state-of-the-art open model under 2B parameters at the time of publication. Stability built the Zephyr variant in three stages. It fine-tuned the base model on seven public instruction datasets, 826,938 samples after dropping conversations longer than eight turns, then aligned the result with direct preference optimisation, which trains on pairs of preferred and rejected answers, using UltraFeedback and Intel’s Orca Pairs and borrowing most hyperparameters from the recipe behind Hugging Face’s Zephyr 7B. A third stage taught the model who made it and what a language model cannot do, using 10,000 opening messages the base model wrote for itself. The variant scores 5.42 on MT-Bench, where a larger model judges answers to open-ended questions, above Qwen 1.5 1.8B Chat at 5.29 and Gemma 2B Instruct at 5.19 and below Stability’s own StableLM Zephyr 3B at 6.64. Chart: Bellagente et al., the authors' own figures. Stability put both sets of weights on Hugging Face along with quantised builds for llama.cpp, Apple’s MLX and Intel’s OpenVINO. The report measures 127 tok/s at 11 W for the 4-bit MLX build on an M2 Mac mini with 8 GB, against 71 tok/s at 6 W in FP16, and 99 tok/s for a 4-bit GGUF build on a 2023 MacBook Pro with 16 GB, while stating these figures are meant as a practical reference rather than rigorous benchmarking. Neither release is open source. The report says the model ships under a non-commercial licence, the Zephyr model card names the Stability AI Non-Commercial Research Community License and points commercial users to the company, and the base model card now carries the Stability AI Community License with commercial use running through a Stability membership. Source: https://arxiv.org/abs/2402.17834 Read the article: https://llmobile.news/ticker/stable-lm-2-zephyr/Stability AIResearchBenchmarksOpen weightsApple Siliconhttps://llmobile.news/ticker/mobile-llms/https://llmobile.news/ticker/mobile-llms/Thu, 22 Feb 2024 19:58:00 +0100Meta Reality Labs researchers published MobileLLM on February 22, 2024, a family of language models built to run on a phone rather than in a data centre. The paper argues that below a billion parameters the shape of the network matters more than the parameter count or the amount of training data. Its 125M and 350M models beat the previous best models at those sizes by 2.7 and 4.3 points on eight zero-shot common sense reasoning tasks. The central finding is that depth beats width. The authors trained 19 models of roughly equal size but different proportions, 9 around 125M parameters and 10 around 350M, and report that 30-layer and 42-layer designs scored higher than the 12-layer shape used by OPT, GPT-Neo and Pythia at the same size. Sharing one weight matrix between the input embedding and the output layer removes 16M parameters, about 11.8% of a 125M model, for a 0.2 point accuracy drop the authors then recover by spending the freed budget on two more layers. Grouped query attention, where several query heads read one shared set of key and value heads, cuts 16 key-value heads to 4 at close to the same accuracy. The paper sets out what a phone can actually spare. The authors put DRAM on current handsets at 6 to 12 GB, shared with the operating system and every other app, and argue that an app should not claim more than 10% of it. For energy they use a rule of thumb of 0.1 J per token per billion parameters, which puts a 7B model at 0.7 J/token, drains 0.2% of an iPhone battery every 64 tokens and allows under 2 hours of conversation at 10 tok/s. A 350M model with 8-bit weights costs 0.035 J/token by the same arithmetic, which the authors say covers a full day of use. Meta also profiled the models on hardware rather than estimating. Running the 125M model in FP16 through ExecuTorch on an iPhone 13 with iOS 17.2.1 and the Metal Performance Shaders backend, the authors measured 39.2 ms to load, 1361.7 ms to initialise and 15.6 ms to execute, the last averaged over 50 runs. MobileLLM-LS, the layer-sharing variant, computes each transformer block twice in a row so that 30 blocks of weights behave like 60 layers at the same file size, a pattern the team picked over the alternatives because the repeated weights stay in the roughly 20 MB of on-chip cache instead of being fetched from DRAM again. That costs 2.2% more loading and initialisation time and 2.6% more execution time, against 143% and 86% for a model that really has 60 layers, and adds 0.7 and 0.8 points of accuracy at 125M and 350M. After fine-tuning, MobileLLM-350M scores 3.28 on the MT-Bench chat benchmark against 1.37 for OPT-350M and 2.24 for the 1.3B OPT, and the layer-shared 350M wins 48.2% of AlpacaEval comparisons against text-davinci-001, a model that wins 50% against itself. For API calling the team built a synthetic set of 5,000 training and 2,500 test conversations that turn requests such as setting an alarm into a function call, and MobileLLM-350M scored 65.3 on intent exact match against 62.8 for Llama 2 7B, while trailing the larger model on the structure of the call and on the wording of the reply. Quantising weights and activations to 8 bits costs under 0.5 points. The work was published at ICML 2024 and the code is in Meta’s repository. Source: https://arxiv.org/abs/2402.14905 Read the article: https://llmobile.news/ticker/mobile-llms/MetaResearchMemoryiPhoneBenchmarkshttps://llmobile.news/ticker/tinyllava/https://llmobile.news/ticker/tinyllava/Thu, 22 Feb 2024 06:05:30 +0100Researchers at Beihang University and Tsinghua University posted TinyLLaVA on February 22, 2024, a framework for building small vision-language models out of an existing language model and an existing image encoder. Their best configuration, which the paper calls TinyLLaVA-3.1B, scores higher than the 7B LLaVA-1.5 on seven of the nine benchmarks in the paper’s comparison table, ties it on GQA at 62.0, and trails it on MME, where the same table gives LLaVA-1.5 1510.7 against 1464.9. The authors report 79.9 against 78.5 on VQAv2 and 75.8 against 63.4 on LLaVA-Bench-in-the-Wild, and put the model ahead of the 7B Qwen-VL on four of the five benchmarks they share, behind it on TextVQA at 59.1 against 63.8. The framework treats four things as variables, the small language model, the vision encoder that turns an image into patch features, the connector that maps those features into the language model’s embedding space, and the training recipe. The authors instantiate it with TinyLlama at 1.1B, StableLM-2 at 1.6B and Phi-2 at 2.7B as language models, CLIP at 0.3B and SigLIP at 0.4B as vision encoders, and either the 558k-caption LLaVA-1.5 data or the 1246k-caption ShareGPT4V data. Their base recipe updates only the connector during pre-training, while their share recipe freezes the first 12 layers of the vision encoder and updates the rest of the model. The ablations put most of the movement in the vision encoder and the recipe. Swapping CLIP for SigLIP raises the Phi-2 variant’s TextVQA score from 51.4 to 55.6 and the TinyLlama variant’s from 45.8 to 49.1, which the authors attribute in part to SigLIP running at 384 pixels against 336 and emitting 729 visual tokens against 576. A two-layer MLP connector beat a resampler on four of six benchmarks in the single comparison the paper runs, 74.0 against 70.1 on VQAv2 and 58.1 against 54.2 on GQA, with the resampler ahead on SQA-I and MM-Vet. The authors report that the share recipe, which fine-tunes part of the vision encoder rather than freezing all of it, improved every variant pre-trained on ShareGPT4V. Model size still helps when data and recipe are held fixed, and with CLIP under the base recipe the Phi-2 variant leads the StableLM-2 and TinyLlama ones on all seven benchmarks in the appendix table. The ordering flips once the recipe changes, with the 1.5B SigLIP TinyLlama model on the share recipe beating the 1.9B CLIP StableLM-2 model on the base recipe on six of seven benchmarks. The authors add a hallucination caveat, reporting that the share recipe lowers POPE accuracy for the StableLM-2 and Phi-2 variants even as their other scores rise, and they conjecture that smaller language models need more trainable parameters during pre-training while larger ones suffer from them. The paper carries no on-device measurement. It states no latency, no memory footprint and no target hardware, and argues the small-scale case from training and inference cost rather than from phone or edge deployment. The authors wrote that weights and code would be made public, and the project repository and the 3.1B weights carry the Apache 2.0 license. Source: https://arxiv.org/abs/2402.14289 Read the article: https://llmobile.news/ticker/tinyllava/ResearchBenchmarksPhiOpen weightsOpen sourcehttps://llmobile.news/ticker/gemma/https://llmobile.news/ticker/gemma/Wed, 21 Feb 2024 18:00:00 +0100Google released Gemma on February 21, 2024, the first two models in the line, at 2B and 7B parameters and in a pretrained and an instruction-tuned checkpoint each. The technical report presents the 7B as a model for deployment on GPU and TPU and the 2B as one for CPU and on-device applications. Google says Gemma was built from the same research and technology used to create Gemini and shares technical and infrastructure components with it. Both sizes are decoder-only transformers trained on a context length of 8192 tokens. The 2B has 18 layers, a model dimension of 2048 and a single key-value head, since Google’s ablations found multi-query attention works well at small scale, while the 7B has 28 layers and keeps standard multi-head attention across 16 heads. Both inherit Gemini’s 256k-entry vocabulary, which puts 524M of the 2B’s parameters into embeddings and leaves 1.98B elsewhere. Google reports training the 2B on 3T tokens and the 7B on 6T, mostly English web documents, mathematics and code. Google reports Gemma 7B at 64.3 on MMLU 5-shot against 54.8 for Llama-2 13B, 46.4 on GSM8K against 28.7, and 32.3 on HumanEval against 18.3, for an average of 56.9 across 18 academic benchmarks. Google puts the 2B at 42.3 on MMLU and 45.0 on average, ahead of Llama-2 7B on the mathematics and coding tasks and behind it overall. Google states that Gemma outperforms similarly sized open models on 11 of the 18 text-based tasks, and notes it could not rerun the Llama-2 evaluations itself because of that model’s licensing, so it cites Meta’s published figures. Google's own figures for Gemma 7B against Llama-2. Chart: Google. The weights went up on Kaggle and Hugging Face, with the models also runnable from Colab and Vertex AI, and Google provided toolchains for inference and supervised fine-tuning across JAX, PyTorch and TensorFlow through native Keras 3.0. Hugging Face added Gemma to Transformers 4.38 on announcement day. llama.cpp merged Gemma support the same day, within half an hour of the pull request opening, which is what brought the 2B into quantised local runs on consumer hardware. Google states that the models run across laptop, desktop, IoT, mobile and cloud. The weights are open but the licence is not a standard open source one. Google publishes them under its own Gemma Terms of Use, which allow use, modification and redistribution provided that downstream recipients get the same terms and the separate Prohibited Use Policy, and which reserve Google’s right to restrict uses it considers non-compliant. Google says the terms permit responsible commercial usage and distribution for all organisations regardless of size. Alongside the models Google shipped a Responsible Generative AI Toolkit with a safety classification method, a tool for debugging model behaviour and written guidance for model builders. Source: https://blog.google/technology/developers/gemma-open-models/ Read the article: https://llmobile.news/ticker/gemma/GoogleGemmaOpen weightsllama.cppBenchmarkshttps://llmobile.news/ticker/mobilevlm-v2/https://llmobile.news/ticker/mobilevlm-v2/Tue, 06 Feb 2024 08:16:36 +0100Researchers at Meituan, Zhejiang University and Dalian University of Technology published MobileVLM V2 on February 6, 2024, a family of vision language models at 1.7B, 3B and 7B parameters, each pairing a CLIP ViT-L/14 encoder that reads images at 336 by 336 pixels with a MobileLLaMA or Vicuna language model. Averaged over six benchmarks covering visual question answering, science questions, reading text in images, object hallucination and two multimodal test suites, the authors score the 1.7B model at 64.2 against 62.7 for LLaVA-1.5 3.3B, and the 3B model at 68.1 against 66.7 for MoE-LLaVA-2.7B×4, a mixture of experts where only part of the model is active per token. Their 7B model reaches 72.1, ahead of ShareGPT4V 7B at 70.8 and LLaVA-1.5 7B at 68.8 on the same average. The authors measured generation over 256 output tokens on an NVIDIA AGX Jetson Orin developer board, not on a phone, with the language model quantised to 4-bit and run through llama.cpp. They report 51.63 tok/s for the 1.7B model, 30.80 tok/s for the 3B and 15.49 tok/s for the 7B, against 43.39, 20.45 and 12.96 tok/s for the three LLaVA-1.5 sizes in the same table. The authors state they wrote their own CUDA implementation of the new projector for that board. Chart: Chu et al., measured on an NVIDIA Jetson Orin with llama.cpp. Most of the speed comes from how much of the image reaches the language model. The encoder turns a 336 by 336 picture into 576 patch tokens, and the projector between the two models, which the authors call LDPv2, first runs two 1 by 1 convolutions to match the language model’s width. It then averages every 2 by 2 block of neighbouring patches into a single token, leaving 144, and finally adds a small depthwise convolution back onto the result so each surviving token keeps a sense of where it sat in the picture. That positional part holds 0.02M parameters against 12.64M for the equivalent block in the first MobileVLM, and the whole projector 6.32M against 18.94M, while the six-benchmark average in the authors’ ablation rises from 62.8 to 64.2. Diagram: Chu et al., MobileVLM V2. Dropping tokens costs accuracy on fine detail. The authors removed the pooling step from the 7B model, which leaves all 576 tokens and puts it at the same latency as ShareGPT4V, and the six-benchmark average rises from 72.1 to 73.5, with most of the gain on TextVQA, a reading task full of small objects. vivo AI Lab took the same 2 by 2 merge further nine months later in BlueLM-V-3B, cutting 729 tokens per image tile to 196 and measuring the result on a phone NPU. The rest of the gain over the first MobileVLM comes from data and from what stays trainable. The authors pretrain on 1.2 million captioned images from ShareGPT4V, then run a second stage over 2.4 million samples drawn from Visual Dialog, TextVQA, COCO Caption, SBU and other academic sets, 3.6 million in total, and they train the projector together with the language model in both stages while the vision encoder stays frozen, where LLaVA-1.5 and the first MobileVLM keep the language model fixed during pretraining. Both stages together took 8 NVIDIA A100 GPUs for about 14 hours. Meituan put weights for all three sizes on Hugging Face the same day, and the code is published under the Apache 2.0 license. Source: https://arxiv.org/abs/2402.03766 Read the article: https://llmobile.news/ticker/mobilevlm-v2/ResearchBenchmarksllama.cppOpen weightsNvidiahttps://llmobile.news/ticker/phi-2/https://llmobile.news/ticker/phi-2/Tue, 12 Dec 2023 18:00:00 +0100Microsoft released Phi-2 on December 12, 2023, a base language model with 2.7 billion parameters. The company says that on complex benchmarks it matches or outperforms models up to 25 times larger, and that it surpasses Mistral and Llama-2 at 7B and 13B parameters on various aggregated benchmarks. Microsoft’s own table scores Phi-2 at 53.7 on grouped coding benchmarks against 38.3 for Llama-2-70B, and at 61.1 on math against 64.1 for the same model. Against Google’s Gemini Nano 2 at 3.2B, Microsoft reports 59.3 versus 42.4 on BigBench-Hard and 59.1 versus 27.2 on MBPP, using Google’s own reported figures for the comparison. Microsoft trained the model on 1.4 trillion tokens from multiple passes over a mixture of synthetic and web data for language and code, following what it calls textbook-quality data. The synthetic sets were written to teach common sense reasoning and general knowledge such as science, daily activities and theory of mind, and the web portion was filtered for educational value and content quality. The model card puts the underlying dataset at 250 billion tokens, combining synthetic text generated with GPT-3.5 and web data from Falcon RefinedWeb and SlimPajama that was assessed with GPT-4. Microsoft did not train Phi-2 from scratch. It started from the 1.3-billion-parameter Phi-1.5 and embedded that model’s knowledge inside the larger one, a step the company says both accelerates training convergence and raises Phi-2’s benchmark scores. Training then took 14 days on 96 A100 GPUs, and the model card lists a context length of 2048 tokens. Phi-2 is a base model that went through neither reinforcement learning from human feedback nor instruction tuning. Chart: Microsoft's own evaluation figures for the two models. Microsoft published the weights in the Azure AI Studio model catalog and on Hugging Face, where the first model card, dated December 13, 2023, named the Microsoft Research License and limited use to research. Microsoft replaced that file on January 5, 2024, and the model card has since stated “The model is licensed under the MIT license”, which the repository’s commit history dates to that day. Source: https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/ Read the article: https://llmobile.news/ticker/phi-2/MicrosoftPhiBenchmarksResearchOpen weightshttps://llmobile.news/ticker/stable-lm-zephyr/https://llmobile.news/ticker/stable-lm-zephyr/Thu, 07 Dec 2023 17:00:00 +0100Stability AI released StableLM Zephyr 3B on December 7, 2023, a chat model with 3 billion parameters that the company aims at edge devices and hardware without the compute of dedicated high-end systems. Stability calls the model 60% smaller than 7B models and reports an MT-Bench score of 6.64 and an AlpacaEval win rate of 76.00%. Both numbers are Stability’s own, measured on two evaluations the company links from the announcement. MT-Bench scores answers to open-ended questions with a larger language model acting as judge, and AlpacaEval measures how often a judge prefers a model’s reply over a reference answer. In the table Stability published with the model card, the 3B model sits above Falcon-Instruct at 40B parameters, which scored 5.17 on MT-Bench, and MPT-Chat at 7B with 5.42, and below Zephyr-7b-beta at 7.34 and Llama2-Chat at 70B with 6.86. Table: Stability AI, the company's own benchmark figures. The model extends Stability’s earlier StableLM 3B-4e1t and borrows its training pipeline from Hugging Face’s Zephyr 7B, according to Stability. Stability first ran supervised fine-tuning on the UltraChat, MetaMathQA, Evol Wizard and Capybara instruction datasets, then aligned the result with direct preference optimisation, which trains a model on pairs of preferred and rejected answers instead of on correct answers alone. That second stage used UltraFeedback from the OpenBMB research group, a set of 64,000 prompts with model responses. Stability put the weights on Hugging Face and linked an example notebook for tuning the model’s speed. The release is not open source. Stability states that it ships under a non-commercial community licence permitting non-commercial use, and tells anyone who wants the model in a commercial product to contact the company for separate terms. Source: https://stability.ai/news/stablelm-zephyr-3b-stability-llm Read the article: https://llmobile.news/ticker/stable-lm-zephyr/Stability AIOpen weightsBenchmarkshttps://llmobile.news/ticker/phi-1-5/https://llmobile.news/ticker/phi-1-5/Mon, 11 Sep 2023 16:00:00 +0200Microsoft Research published phi-1.5 on September 11, 2023, a 1.3-billion-parameter model that applies the synthetic textbook data approach of phi-1 to common sense reasoning and general world knowledge rather than Python code. The authors report common sense benchmark results comparable to models ten times the size trained on more than ten times as much data, and describe natural language performance as comparable to models five times larger. phi-1.5 scores 0.734 on WinoGrande against 0.691 for Llama2-7B, 0.444 on ARC-Challenge against 0.434, and 0.526 on SIQA against 0.480, all measured in Microsoft’s own evaluation pipeline. The gap widens on multi-step reasoning, with 40.2 on GSM8K grade-school maths against 14.6 for Llama2-7B, 34.1 on HumanEval against 12.8, and 37.7 on MBPP against 20.8, which the authors note puts its coding scores at or above Llama-65B. MMLU runs the other way, at 0.376 for phi-1.5 and 0.453 for Llama2-7B. Chart: Microsoft, Figure 1 of the technical report. The training set combines 7B tokens from phi-1 with roughly 20B tokens of newly generated textbook-like text, seeded from 20,000 selected topics spanning science, daily activities and theory of mind. That yields a 30B-token dataset whose only non-synthetic part is 6B tokens of filtered code, and training runs over 150B tokens, 80 percent of them from the new synthetic data. Microsoft puts the cost at 1,500 GPU hours on a single A100-80G, against more than 80,000 hours for Llama-7B. The same report covers two variants built to test what web data adds. phi-1.5-web-only trains purely on 95B tokens of filtered web text, 88B of it from the Falcon RefinedWeb dataset and 7B of code from The Stack and Stack Overflow, while phi-1.5-web mixes filtered web data, phi-1’s code and the synthetic text in shares of roughly 40, 20 and 40 percent. phi-1.5-web comes out ahead on reasoning at 44.6 on GSM8K, 41.4 on HumanEval and 43.5 on MBPP. The authors list hallucinations and the potential for toxic and biased generations among the traits phi-1.5 shares with larger models, and write that it “is not immune” despite the absence of web data in its training set. On an 86-prompt probe they wrote themselves, phi-1.5 passed 47 and failed 34, while Llama2-7B and Falcon-7B failed 54 and 50. Microsoft released the raw base model with no instruction fine-tuning and no RLHF, and the weights sit on Hugging Face under the MIT license. Source: https://arxiv.org/abs/2309.05463 Read the article: https://llmobile.news/ticker/phi-1-5/MicrosoftPhiResearchBenchmarksOpen weightshttps://llmobile.news/ticker/phi-1/https://llmobile.news/ticker/phi-1/Tue, 20 Jun 2023 18:14:00 +0200Microsoft Research published phi-1 on June 20, 2023, a Python code model with 1.3B parameters. The paper reports 50.6 percent pass@1 on HumanEval and 55.5 percent on MBPP, pass@1 being the share of problems the model solves with its first generated answer. The same table lists StarCoder at 15.5B parameters and 1 trillion training tokens with 33.6 percent and 52.7 percent, GPT-3.5 at 175B parameters with 47 percent, and CodeGen-Mono at 16.1B parameters and 577B tokens with 29.3 percent. The training data comes to under 7B tokens in three parts. About 6B tokens are Python files pulled from The Stack and StackOverflow by a classifier the authors trained on roughly 100,000 GPT-4 ratings of how instructive a snippet is for someone learning to code. Under 1B tokens are synthetic Python textbooks written by GPT-3.5, and about 180M tokens are synthetic exercises with solutions, used only in the finetuning step that turns phi-1-base into phi-1. Microsoft ran the training on 8 Nvidia A100 GPUs, reaching the base model in under 4 days and spending another 7 hours on the finetuning. That amounts to about 8 passes over the pretraining data and a little over 50B tokens seen, which the paper’s chart puts at 1090 GPU hours for the 1.3B run. Chart: Microsoft, Figure 2.1 of the paper. The GPU hours on the axis are the authors' own figures. The same pipeline at 350M parameters yields phi-1-small, which the authors put at 45 percent on HumanEval and use to argue that parameter count still drives which capabilities appear at all. Microsoft posted phi-1 on Hugging Face in September 2023, where the model card lists the MIT license. Source: https://arxiv.org/abs/2306.11644 Read the article: https://llmobile.news/ticker/phi-1/MicrosoftPhiResearchBenchmarksOpen weightshttps://llmobile.news/ticker/lamini-lm/https://llmobile.news/ticker/lamini-lm/Thu, 27 Apr 2023 19:58:00 +0200Researchers at MBZUAI, the University of British Columbia and Monash University published LaMini-LM on April 27, 2023, a family of 17 distilled instruction-following models ranging from 61M to 7B parameters. The paper gives resource use as the motivation, stating that earlier distilled models of 7B to 13B parameters present “challenges for deployment in resource-constrained settings” and that the authors set out to make deployment in such settings easier. The paper names no target device and reports no memory footprint, no quantisation and no generation speed, and the authors state they trained and evaluated on V100 and A100 GPUs. The models are fine-tuned on the LaMini instruction dataset, 2.58M instruction and response pairs that the team generated with gpt-3.5-turbo from prompts taken from self-instruct, P3, FLAN and Alpaca. The authors describe it as the largest instruction dataset of its kind at the time and 50 times the size of the dataset behind Alpaca. The students are 6 encoder-decoder models built on T5 and Flan-T5 and 11 decoder-only models built on GPT-2, GPT-Neo, Cerebras-GPT, GPT-J and LLaMA. The authors evaluated the family on 15 NLP benchmarks with EleutherAI’s lm-evaluation-harness and report that LaMini-Flan-T5-248M performs on par with LLaMA-7B on the benchmark average, against a baseline with about 28 times as many parameters. They warn in the repository that their LLaMA figures are not comparable to the ones the LLaMA authors published, because they re-ran the evaluation themselves. A human rating of 114 user-oriented instructions follows the same size trend and leaves a gap to gpt-3.5-turbo, and the paper reports that the models do poorly on coding, maths and tasks that need reasoning. MBZUAI publishes the weights and the instruction dataset on Hugging Face. The repository puts the code under Apache 2.0 and the models and the data under CC BY-NC 4.0, and states that both are intended for research use only. The paper was later accepted at the EACL 2024 main conference. Source: https://arxiv.org/abs/2304.14402 Read the article: https://llmobile.news/ticker/lamini-lm/ResearchDistillationOpen weightsBenchmarks