https://llmobile.news/tags/open-source/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.news/ticker/online-sdft/https://llmobile.news/ticker/online-sdft/Tue, 18 Aug 2026 16:57:46 +0200I-Ju Lin and Zhang-Wei Hong published Online-SDFT, a method that keeps fine-tuning a small language model on a phone from the user’s own stream of decisions. On a synthetic notification-routing benchmark with Liquid AI’s LFM2.5-230M, the authors report 70.28% of decisions matching a hidden user preference against 52.78% for the strongest baseline they tested, and cumulative regret of 44.76 against 105.26, over 720 decisions in three paired streams. They state that this comparison ran on Apple MPS rather than on a phone, and that the confidence intervals are nominal across only three streams. Self-distillation here means the model teaches itself. The model picks an action first, the phone later observes what happened, and the same model is then run again with that outcome added to its input. Its better-informed answer becomes the training target for the version that saw only the original context, so no second teacher model, no human label and no reward score is needed. The authors keep the 230M base frozen and train a LoRA adapter of 172,032 parameters across 48 tensors in six attention layers, with the teacher pass running with the adapter switched off so student updates cannot move the teacher. Feeding one interaction at a time into so small an adapter makes it noisy and prone to forgetting, so the authors keep a 64-row buffer of recent lessons and sample each batch from it. Their ablation puts that buffer ahead of the exploration schedule as the load-bearing part, with accuracy dropping from 70.28% to 39.58% and regret climbing from 44.76 to 134.90 once replay is removed, while dropping exploration instead leaves accuracy at 69.44% and regret at 62.00. Diagram: I-Ju Lin and Zhang-Wei Hong. The repository carries an Android proof of concept in which LFM2.5-230M decides and ONNX Runtime Training performs the adapter update on the device, with replay and checkpoints in app-private storage that survive a restart. Exporting the model still happens on a Linux host, and the deployment guide asks for Android 12 or newer on ARM64 with 12 GB of RAM at minimum, 16 GB preferred, and 8 GB of free storage, because the training graph is FP32. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Video: I-Ju Lin and Zhang-Wei Hong. Watch on the project page The authors call that graph a correctness implementation rather than a claim of practical background training on low-memory phones, and state that they did not measure Android latency, peak memory, battery use or thermal behaviour, and that the benchmark uses synthetic streams rather than real users. The code, the Android app and a Colab notebook are on GitHub under the MIT license. No paper or preprint accompanies the release. Source: https://lin826.github.io/SLM-Online-SDFT/ Read the article: https://llmobile.news/ticker/online-sdft/ResearchDistillationAndroidLFMOpen sourcehttps://llmobile.news/ticker/rikkahub-agent-local-llm/https://llmobile.news/ticker/rikkahub-agent-local-llm/Sat, 08 Aug 2026 00:00:00 +0200Adam Conway at XDA has tested RikkaHub Agent, a fork by ExTV of the open-source Android LLM client RikkaHub, which adds an agent layer with more than 80 device tools. The phone runs the agent loop and executes the tools; the model itself runs elsewhere on the local network over an OpenAI-compatible endpoint. The test device was an Oppo Find N5 with a Snapdragon 8 Elite and 16 GB of RAM. Inference ran on a Radeon RX 7900 XTX with Qwen 3.6 27B at Q4_K_M, roughly 16 GB, at 40 to 50 tokens per second. A second configuration used Qwen 3.6 35B on a Lenovo ThinkStation PGX with Nvidia’s GB10 Grace Blackwell. Screenshot: XDA. The tool set covers tapping, swiping, scrolling and typing on screen, reading notifications, sending SMS, file management, browser control, SSH, scheduled jobs, a JavaScript runtime and Telegram. All tools are off by default and enabled per assistant, actions with side effects require approval, and a deterministic blocklist rejects destructive commands. The default context is 7,000 input tokens, rising to 32,000 with every tool enabled. Asked whether whisper.cpp was installed, the agent determined that neither the command-line tool nor a model was present, requested permission, then installed the dependencies, cloned the repository and compiled it inside Termux. The build took seven minutes, and the agent chose the 75 MB small model on speed and accuracy grounds. It then transcribed a Telegram voice message, installing ffmpeg, converting the OGG/Opus file to 16 kHz mono WAV and returning a transcript in about two seconds. Running the model on the phone itself was slower. With Gemma 4 E2B, a 2.6 GB build accelerated by LiteRT-LM, Conway reports chat worked but agentic tasks were painful, and tool calls were emitted as plain text until an update fixed the behaviour. Frontier intelligence is never what I’d want from a phone agent, which is why a local LLM works perfectly. Source: https://www.xda-developers.com/turned-phone-ai-agent-powered-local-llm-installed-software/ Read the article: https://llmobile.news/ticker/rikkahub-agent-local-llm/AndroidAgentsQwenOpen sourceSpeechhttps://llmobile.news/ticker/airgap/https://llmobile.news/ticker/airgap/Wed, 29 Apr 2026 08:06:11 +0200Xavier Puspus published Airgap on April 29, 2026, a React Native starter kit for branded customer support apps that have to keep answering when the network is slow, missing or deliberately blocked. Its configuration file pins a Gemma 4 E2B instruction-tuned model in Q3_K_S GGUF form, about 2.4 GB, which the app downloads on first launch and runs through llama.rn, the React Native binding for llama.cpp. The README states that Airgap searches bundled local documents first and uses a model only to phrase what that retrieval returned, and it lists Apple Foundation Models and Google’s ML Kit GenAI Prompt API as two further on-device answer providers an operator can put ahead of the downloaded file. The repository ships seven example configurations, for airline, banking, electric utility, healthcare, insurance, telecom and water utility support, each with its own branding, prompts and JSON knowledge files, and the downloaded-model path targets iOS 15.1 and newer and Android API 24 and newer. Airgap is published under the MIT license. It is an early single-author project, with 54 commits and no tagged release, and almost all of the code arrived in one squashed initial commit followed by a burst of work in August 2026. The README lists physical-device evaluation of the Apple and Android system providers as an open release gate and labels the published demo recordings as Android emulator and iOS simulator footage rather than evidence of either system model running on a phone. Source: https://github.com/xmpuspus/airgap Read the article: https://llmobile.news/ticker/airgap/Open sourceAndroidiOSGemmallama.cpphttps://llmobile.news/ticker/clawmobile/https://llmobile.news/ticker/clawmobile/Thu, 26 Feb 2026 13:34:00 +0100Seven researchers at MBZUAI, City University of Hong Kong and one working independently published ClawMobile on February 26, 2026, an agent runtime that runs on the Android phone itself with no tethering to a host machine. The model is not on the device. The authors state that ClawMobile runs locally while model inference is performed remotely, and all agents in their tests use GPT-5.2, so the split is an on-device runtime driving a cloud model. Across six real-life phone tasks the authors report ClawMobile completing all six at 100 percent, against 33 to 100 percent for the UI agent they compare against. What separates it from screen-reading GUI agents is that the reasoning loop never touches the screen. The orchestrator issues tool calls to control backends instead, and the authors describe a deterministic-first policy in which the runtime first checks its memory for a structured interface, reaching for ADB system commands and the Termux hardware API before anything visual. A semantic UI agent is invoked only when a step genuinely depends on reading dynamic interface state, and raw taps on interface elements sit below that as a fallback. After every action the runtime re-queries device state to confirm the step landed, which the authors contrast with inferring execution success purely through model reasoning. Architecture diagram: Du et al., CC BY 4.0. The tests ran on a Google Pixel 9 on Android 16, with DroidRun as the UI agent baseline and a stripped ClawMobile without DroidRun as a second baseline, all three on GPT-5.2 and scored by human annotators. In the authors’ table DroidRun reaches 100 percent on switching the system to dark theme and on playing a YouTube video, 85 percent on posting a YouTube comment, 73 percent on a Chrome search and on a cross-app task that searches football results and writes them into Notes, and 33 percent on installing an app from the Play Store, where ClawMobile scores 100 percent throughout. The authors put the cost of that at 57.5 seconds slower per task on average, with ClawMobile taking 21 seconds for the dark theme switch and 235 seconds for the YouTube comment. Their deterministic-only variant hits the 600-second timeout on both YouTube tasks. The authors name the remote model as the main open problem, writing that fully remote inference introduces network latency and privacy concerns and calling on-device inference a promising direction they have not taken. They also flag that serialising full UI trees or screenshots into model context can dominate token use and latency on mobile, and present the results as preliminary, six tasks on one device. The code is open source on GitHub, built on the OpenClaw agent framework with a Telegram bot as the chat channel, and the paper was accepted at EuroMLSys 2026 under a Creative Commons Attribution 4.0 license. Source: https://arxiv.org/abs/2602.22942 Read the article: https://llmobile.news/ticker/clawmobile/AgentsAndroidPixelResearchOpen sourcehttps://llmobile.news/ticker/apple-foundation-models-sdk/https://llmobile.news/ticker/apple-foundation-models-sdk/Wed, 25 Feb 2026 20:47:00 +0100Apple published python-apple-fm-sdk, Python bindings for its Foundation Models framework, with a first beta release on February 25, 2026 according to the repository’s release history. The package reaches the on-device model at the core of Apple Intelligence from Python on macOS, and installs with pip install apple-fm-sdk. The Python API follows the Swift one. SystemLanguageModel reports whether the model is available and, if not, why, while LanguageModelSession holds a conversation and its async respond call returns text or streams it. A @fm.generable decorator marks a Python class for the model to fill in, and fm.guide adds per-field constraints such as a numeric range, as the README shows with a Cat class whose age is bounded to 0 through 20. Tool calling and session transcripts are exposed as well. The bindings do not reimplement inference. The repository carries a Swift package named foundation-models-c that wraps the framework behind a C interface and builds it as a dynamic library, which the Python layer then calls. Apple writes in the documentation that the bindings run the Swift framework underneath, so evaluations reflect real on-device performance and behaviour, and its evaluation guide notes that inference calls are processed one at a time rather than in parallel at the macOS hardware level. Apple lists macOS 26.0 or later, Xcode 26.0 or later with its agreement accepted in the Xcode app, Python 3.10 or later, and Apple Intelligence switched on. PyPI carries only source distributions for the package, so the Swift component is compiled locally during installation, which is what the Xcode requirement covers. Apple Intelligence on the Mac runs on Apple silicon machines only. The code is under the Apache 2.0 license and the package metadata on PyPI classifies it as alpha. Apple pushed 0.1.0 to PyPI on March 8, 2026, added the Attachment API for sending images alongside text in 0.2.0 on June 8, 2026 for WWDC 2026, and exposed the model’s context size and the token count of a given input in 0.2.1 on June 29, 2026. The README states the project is not yet taking contributions. Source: https://github.com/apple/python-apple-fm-sdk Read the article: https://llmobile.news/ticker/apple-foundation-models-sdk/AppleDeveloper toolsApple SiliconOpen sourcehttps://llmobile.news/ticker/off-grid-offline-ai-app/https://llmobile.news/ticker/off-grid-offline-ai-app/Sat, 14 Feb 2026 23:39:00 +0100Ali Chherawalla has posted Off Grid to Show HN, an Android and iOS app that runs text generation, image generation, vision and speech recognition entirely on the device. The submission reached 124 points and 66 comments. The repository is licensed under MIT. The app bundles llama.cpp for text, Stable Diffusion for images and whisper.cpp for speech to text. It accepts any GGUF file and lists Qwen 3, Llama 3.2, Gemma 3 and Phi-4 among supported models; vision runs on SmolVLM, Qwen3-VL or Gemma 3n. It also connects to OpenAI-compatible servers on a local network, and builds a project knowledge base from PDFs chunked and embedded on-device with a bundled MiniLM model into SQLite. The README reports 15 to 30 tokens per second on flagship CPUs and 20 to 40 tokens per second through Adreno GPUs via OpenCL on a Snapdragon 8 Gen 2 or newer. Image generation is given as 5 to 10 seconds on the Hexagon NPU and vision inference at about 7 seconds. The NPU path is marked experimental: it accelerates only Q4_0 and Q8_0 quantisation, and K-quants fall back to the CPU. Comments disputed whether models of that size are usable. One reader wrote: Ive tried a lot of self hosted models at a lot of sizes; those small models are not good enough, and do not have a context long enough to be useful for most everyday operations. Chherawalla replied that “you don’t need claude level performance for a lot of day to day tasks” and that Qwen3 8B, or 4B quantised, is “actually quite good”. Other threads compared the app with PocketPal AI, SmolChat and Google’s AI Edge Gallery, asked for an F-Droid build, and reported that image generation required network access on GrapheneOS. A reader on a Samsung device reported the interface did not move for the on-screen keyboard; the developer shipped a fix the same day. Source: https://news.ycombinator.com/item?id=47019133 Read the article: https://llmobile.news/ticker/off-grid-offline-ai-app/AndroidiOSOpen sourcellama.cppImage generationhttps://llmobile.news/ticker/executorch-1-0/https://llmobile.news/ticker/executorch-1-0/Wed, 22 Oct 2025 18:00:00 +0200The PyTorch team released ExecuTorch 1.0 on October 22, 2025, the generally available version of its runtime for PyTorch models on phones, laptops and embedded devices. It takes a model straight from PyTorch without converting it to ONNX or TFLite and without rewriting it in another language, which the team says removes a common source of numerical mismatches and lost debug information. The beta arrived in October 2024. Diagram: PyTorch documentation. Five backends moved from beta to production status. XNNPACK with Arm Kleidi covers CPU, Apple Core ML covers Apple silicon, the Qualcomm AI Engine delegate covers the Hexagon NPU, and Arm Ethos-U and Vulkan cover NPU and GPU. New backends arrived for Arm VGF, the NXP eIQ Neutron NPU, Samsung Exynos NPU and GPU, and Intel OpenVINO. Meta says features built on ExecuTorch already run on Instagram, WhatsApp, Messenger and Facebook. Anuj Kumar, Director at Reality Labs, describes what it drives in Meta’s wearables, including the Ray-Ban Display glasses with EMG band. Advanced AI features like speech recognition, motion sensing, and computer vision run on these devices by leveraging cutting-edge hardware. Lysandre Debut, Chief Open-Source Officer at Hugging Face, says more than 80 percent of the most downloaded edge-friendly language models on Hugging Face run on ExecuTorch out of the box. The team validated coverage across object detection, depth, OCR, speech recognition and segmentation as well as text and multimodal models such as Voxtral for audio and Gemma 3 for image and text input. The main branch also allows embedding ExecuTorch into native C++ desktop applications. Source: https://pytorch.org/blog/introducing-executorch-1-0/ Read the article: https://llmobile.news/ticker/executorch-1-0/MetaPyTorchDeveloper toolsNPUOpen sourcehttps://llmobile.news/ticker/smollm3/https://llmobile.news/ticker/smollm3/Tue, 08 Jul 2025 12:30:00 +0200Hugging Face released SmolLM3 on July 8, 2025, a 3B model trained on 11.2 trillion tokens that takes up to 128k tokens of context, covers six languages and answers in either of two modes. It is the successor to SmolLM2, whose largest model was 1.7B with an 8k context window and, in Hugging Face’s own words, primarily understands and generates content in English. The longer context comes from an architecture change plus a separate training stage. Hugging Face removed rotary position embeddings from every fourth layer, a technique it calls NoPE, and replaced multi-head attention with grouped-query attention to shrink the key-value cache during inference. After the main pretraining run it trained the model on another 100 billion tokens in two steps, 4k to 32k and then 32k to 64k, and YaRN, a method for extrapolating past the trained length, carries it to 128k tokens at inference, though the published configuration ships set to 64k and has to be edited to go further. Multilingual text made up 12 percent of the web data at every pretraining stage, and Hugging Face lists English, French, Spanish, German, Italian and Portuguese as supported. Hugging Face’s own base-model figures put SmolLM3 at 76.2 on HellaSwag, 65.6 on ARC and 79.0 on BoolQ, ahead of both Llama 3.2 3B and Qwen2.5 3B and ahead of the larger Qwen3 4B on all three. Qwen3 4B stays in front on maths and code, at 51.2 against 46.1 on MATH and 54.9 against 30.5 on HumanEval+, and on the Ruler 64k long-context test Llama 3.2 3B leads at 72.9 against 67.9 for SmolLM3. On the Flores-200 translation benchmark SmolLM3 scores 56.3 against 55.8 for Qwen3 4B and 54.9 for Qwen2.5 3B, and its multilingual chart shows it ahead of both 3B rivals in all five non-English languages and behind Qwen3 4B in all five. Table: Hugging Face's own base-model figures. A flag in the system prompt, /think or /no_think, decides whether the instruct model reasons before it answers. Hugging Face reports the reasoning mode at 36.7 percent on the AIME 2025 maths competition against 9.3 percent without it, 41.7 against 35.7 on GPQA Diamond and 30.5 against 15.2 on LiveCodeBench v4. Qwen3 4B leads on most of the eight tests in that comparison in both modes, at 58.8 on AIME 2025 and 52.9 on LiveCodeBench, though SmolLM3 without reasoning tops it on the IFEval instruction-following test at 76.7 to 68.9. Table: Hugging Face's own instruct-model figures. Hugging Face published the recipe as well as the weights. Pretraining ran in three stages that moved web data from 85 percent to 75 and then 63 percent while code rose from 12 percent to 24 and maths from 3 percent to 13, each share drawn from named public datasets, and the company released the training configs with the exact data weights along with intermediate checkpoints. Both the base and instruct models are on Hugging Face under the Apache 2.0 license, and the model card names llama.cpp, ONNX, MLX, MLC and ExecuTorch for local inference. The 4-bit GGUF published by ggml-org on release day is 1.92 GB, against 3.27 GB for the 8-bit build for MLX and 6.15 GB for the bfloat16 weights. Diagram: Hugging Face. Source: https://huggingface.co/blog/smollm3 Read the article: https://llmobile.news/ticker/smollm3/Hugging FaceOpen weightsBenchmarksOpen sourcellama.cpphttps://llmobile.news/ticker/googleai-edge-gallery/https://llmobile.news/ticker/googleai-edge-gallery/Sun, 18 May 2025 00:21:00 +0200Google’s AI Edge team has published Google AI Edge Gallery, an app that runs generative models entirely on the device. The home screen offers three tasks, AI Chat for multi-turn conversation, Ask Image for questions about a photo from the camera or the gallery, and Prompt Lab for single-turn work, which covers free-form prompts, rewriting text in a chosen tone, summarising and generating code. Once a model file is on the phone, the app needs no connection and Google states that all processing happens on the device. Screenshot: Google. The third screen shows the per-model settings, including the GPU and CPU toggle. Models come from the LiteRT Community organisation on Hugging Face, which the app’s welcome text links to directly. The first release listed three downloads, Gemma 3 1B IT at 4-bit in 554.7 MB, Hammer 2.1 1.5B at 8-bit and Qwen2.5 1.5B Instruct at 8-bit, the latter two at roughly 1.6 GB each. Gated repositories ask for a Hugging Face access token, which the settings dialog stores and can clear again. Anything outside that list has to arrive as a LiteRT .task bundle picked from device storage, and the app tells the user that no other file type is supported. Each model then carries a settings dialog with an accelerator switch between GPU and CPU, next to max tokens, top-k, top-p and temperature. The shipped list sets a different default per model, GPU first for Gemma 3 1B and CPU only for Qwen2.5 1.5B Instruct. After every reply the app prints four of its own measurements, time to first token in seconds, prefill speed and decode speed in tokens per second, and total latency in seconds. It derives them from the session itself, dividing the prompt’s token count by the time the first token took and counting decoded tokens against the clock that starts with that first token, so the numbers describe whatever hardware the app happens to be running on. Google calls this an “experimental Alpha release” and ships it under the Apache 2.0 licence. There is no store listing at this point, so installing means taking the APK from the project’s releases page on a device running Android 8.0 or newer, with a project wiki covering the corporate-device case. The README lists Android as available now and iOS as coming soon. Underneath, the app builds on LiteRT and Google’s MediaPipe LLM inference API for Android. Source: https://github.com/google-ai-edge/gallery Read the article: https://llmobile.news/ticker/googleai-edge-gallery/GoogleAndroidLiteRTHugging FaceOpen sourcehttps://llmobile.news/ticker/flower-intelligence/https://llmobile.news/ticker/flower-intelligence/Wed, 12 Mar 2025 10:00:00 +0100Flower Labs released Flower Intelligence on March 12, 2025 as a preview, an open source inference library that runs a language model on the user’s own device and only moves the work to a remote service if the app and the user allow it. The company shipped SDKs for Swift and TypeScript, and the TypeScript build defaults to Llama 3.2 1B in 16-bit precision. The library picks a local engine from the environment it finds itself in. The TypeScript package depends on MLC’s WebLLM, which executes models in the browser through WebGPU, and on Hugging Face transformers.js, which it uses when the code runs under Node.js rather than in a page. The Swift package is built on MLX Swift and Hugging Face swift-transformers, and declares iOS 16 and macOS 14 as its minimum platforms, so local inference on Apple hardware goes through MLX. Flower Labs’ model reference covers Llama 3.2 at 1B and 3B, Llama 3.1 8B, SmolLM2 at 135M, 360M and 1.7B, and DeepSeek R1 Distill Llama 8B, most of them as 4-bit builds. Which of those run on device depends on the backend. Node.js is limited to the 1B and SmolLM2 entries, while the 8B models run only in the browser and on the MLX Swift path, and Llama 3.2 3B in 16-bit is MLX Swift only. Remote handoff is off by default. An application has to set the remoteHandoff property and supply an API key before the library will contact Flower Confidential Remote Compute, and even with both in place a chat call tries a local engine first and retries remotely only when no local engine matches the model or the local run fails. Individual requests can be pinned with forceLocal or forceRemote. Flower Labs describes the remote service as a private extension of the device that costs the developer no extra work, and named Mozilla Thunderbird as an early adopter building its Thunderbird Assist feature on the library. Ryan Sipes, Managing Director for Mozilla Thunderbird, tied that choice to the mail client’s user base in the announcement. Our 20 million users expect data privacy from every feature we build. Both SDKs are published under the Apache 2.0 license, the TypeScript one as @flwr/flwr on npm and the Swift one as a package in the Flower repository. Flower Labs called the release inference-only and listed fine-tuning, RAG and pre-training as later steps, with the remote compute service available on application for early access. Source: https://flower.ai/blog/2025-03-12-flower-intelligence Read the article: https://llmobile.news/ticker/flower-intelligence/Open sourceDeveloper toolsLlamaWebGPUApple Siliconhttps://llmobile.news/ticker/alibaba-mnn/https://llmobile.news/ticker/alibaba-mnn/Thu, 23 Jan 2025 13:30:00 +0100Alibaba published the first public build of its MNN-LLM Android app on January 23, 2025, a client that downloads open models and runs them entirely on the phone, covering text-to-text, image-to-text, audio-to-text and text-to-image generation through diffusion models. It sits on MNN, the inference and training framework Alibaba has kept open source since 2019 and uses in more than 30 of its own apps, among them Taobao, Youku and DingTalk. Alibaba reports the runtime prefilling 8.6 times faster than llama.cpp on an Android CPU, with decoding 2.3 times faster. Models reach the device through llmexport, which converts a PyTorch checkpoint to ONNX and then to MNN’s own format, splitting the weights into a separate file and quantising them to 4-bit or 8-bit by default, with block size configurable and 5-bit and 6-bit available through the MNNConvert tool. The same README lists Qwen, Gemma, Llama including TinyLlama and MobileLLM, Baichuan, Yi, DeepSeek, InternLM, Phi, ReaderLM and SmolLM as supported families. LoRA adapters can be merged into the base model at export or kept as a separate file that the runtime loads and swaps while running. The MNN-LLM paper, presented at the ACM Multimedia Asia 2024 workshops, describes a DRAM-Flash split that leaves the embedding table, roughly 15% of the weights, in bfloat16 on flash storage and reads only the single token needed for each decode step. The authors put the saving for Qwen2 7B at about 2.18 GB of DRAM. Layers and the LM head stay in memory at int4 or int8 with asymmetric quantisation, computed as W4A8 or W8A8 on CPU and W4A16 or W8A16 on GPU, while the KV cache quantises keys to int4 or int8 and values to fp8 before spilling to flash with prefetching. The team measured the speed comparisons on a Xiaomi 14 against llama.cpp, MLC-LLM and fastllm, using Qwen2 1.5B, Qwen2 7B and Llama 3 8B. Android builds run on the CPU with ARMv8.2 fp16 instructions and add an OpenCL path for the GPU, and the same runtime compiles for iOS with Metal, for macOS, Linux and Windows, and for browsers through WebAssembly. Alibaba distributes the app as a downloadable APK and warns that the first version was tested only on a OnePlus 13 and a Xiaomi 14 Ultra, so low-spec phones may run slowly or not at all. MNN is published under the Apache 2.0 license. Diagram: Alibaba. The architecture overview from the MNN repository. Source: https://github.com/alibaba/MNN Read the article: https://llmobile.news/ticker/alibaba-mnn/AlibabaAndroidQuantisationQwenOpen sourcehttps://llmobile.news/ticker/smollm2/https://llmobile.news/ticker/smollm2/Thu, 31 Oct 2024 19:30:00 +0100Hugging Face released SmolLM2 on October 31, 2024, a second generation of its small language models at 135M, 360M and 1.7B parameters. The company trained the 1.7B model on 11 trillion tokens, against 1T for the first SmolLM at the same size, and its model cards put the 360M model at 4T tokens and the 135M at 2T. The mixture combines FineWeb-Edu, DCLM and The Stack with datasets Hugging Face curated for this run, which its paper from February 2025 names as FineMath for mathematics, Stack-Edu for code and SmolTalk for instruction tuning. That paper states the 1.7B run used a multi-stage schedule that held the highest quality maths and code data back for the final tenth of training, from 10T to 11T tokens, while the 360M and 135M models trained in a single stage on a mix kept consistent throughout. Hugging Face fine-tuned the instruct versions on SmolTalk and then tuned them further on ranked answer pairs from UltraFeedback. Hugging Face reports the 1.7B base model at 60.5 on ARC against 49.2 for Llama 3.2 1B and 58.5 for Qwen2.5 1.5B, 36.7 on TriviaQA against 28.1 and 20.9, and 19.4 on MMLU-Pro against 11.7 and 13.7, all run zero-shot with its own lighteval harness and printed in the model card. The instruct version scores 56.7 on the IFEval instruction-following test against 53.5 for Llama 3.2 1B Instruct and 47.4 for Qwen2.5 1.5B Instruct, while Qwen2.5 stays ahead on the MT-Bench chat test at 6.52 to 6.13. At the smaller sizes Hugging Face compares the 360M model with Qwen2.5 0.5B, reporting 53.0 against 45.4 on ARC and 16.9 against 4.3 on TriviaQA, and publishes no third-party comparison for the 135M model. Table: Hugging Face's own figures for the base models. The weight files run 269 MB for the 135M model, 724 MB for the 360M and 3.42 GB for the 1.7B in bfloat16, and Hugging Face uploaded 4-bit ONNX builds the same day at 118 MB, 273 MB and 1.11 GB, plus a 4-bit GGUF of the 1.7B instruct model at 1.06 GB. The repository says the models run locally through llama.cpp, MLX, MLC and transformers.js. All six checkpoints carry an 8k context window and the Apache 2.0 license. Update, November 27, 2024. Hugging Face published a browser chat demo that runs the 1.7B instruct model through transformers.js and ONNX Runtime Web on WebGPU, with no server call after the download. Its local inference guide adds runnable code for llama.cpp, MLX and MLC, and lists the models on Ollama and in the PocketPal-AI phone app. Source: https://github.com/huggingface/smollm Read the article: https://llmobile.news/ticker/smollm2/Open weightsBenchmarksOpen sourcellama.cppWebGPUhttps://llmobile.news/ticker/amd-llama-135m/https://llmobile.news/ticker/amd-llama-135m/Fri, 27 Sep 2024 19:00:00 +0200AMD released AMD-Llama-135M on September 27, 2024, its first small language model, at 135M parameters and trained from scratch on AMD’s own accelerators. AMD states that pretraining used 670B tokens drawn from SlimPajama and Project Gutenberg and took six days on four Instinct MI250 nodes holding four MI250 accelerators each. A second variant, AMD-Llama-135M-code, adds 20B tokens from the Python subset of StarCoder over another four days on the same hardware. That code variant exists to work as a draft model for speculative decoding, where a small model guesses several tokens in one pass and a larger model then checks them, so the larger model can accept several tokens per forward pass rather than producing one at a time. AMD paired it with Meta’s CodeLlama-7b and measured throughput on the HumanEval coding benchmark. AMD reports about 2.8x higher throughput on an Instinct MI250 accelerator, about 3.88x on the CPU of a Ryzen AI processor, and about 2.98x with the 7B model running 4-bit on the Ryzen AI NPU while the draft model ran on the CPU, each against the same setup without speculative decoding. AMD’s footnotes name a Ryzen 9 PRO 7940HS as the test machine for the Ryzen AI numbers. Diagram: AMD, showing its own measured speedups. AMD published the weights on Hugging Face and the training, dataset and inference code on GitHub, under the Apache 2.0 license. AMD states the training code builds on TinyLlama and uses PyTorch FSDP for distributed training across nodes. Source: https://www.amd.com/en/developer/resources/technical-articles/introducing-amd-first-slm-135m-model-fuels-ai-advancements.html Read the article: https://llmobile.news/ticker/amd-llama-135m/AMDLlamaNPUOpen weightsOpen sourcehttps://llmobile.news/ticker/on-device-collaborative-lm/https://llmobile.news/ticker/on-device-collaborative-lm/Sat, 21 Sep 2024 00:34:00 +0200Four researchers at EPFL published CoMiGS on September 20, 2024, a scheme in which many devices fine-tune a language model together, with some of the added model parts shared across everyone and the rest kept on the device that trained them. In a simulation of four users, each holding Wikipedia articles in a different language, the authors measure test perplexity at 47.19 with GPT-2 124M as the base model, against 54.38 when each user trains alone on local data and 58.80 for federated averaging, which pools every user’s update into one shared model. Perplexity measures how surprised a model is by the next token, so lower is better. The paper appeared at ICML 2025. Nobody retrains the base model. Each device trains small LoRA adapters on top of pretrained weights that stay frozen, and CoMiGS sorts those adapters into two kinds. Generalist adapters are averaged across all devices at every communication round, specialist adapters and a small router never leave the device, and the router decides per token which adapters to use. The authors train that router against a separate validation set rather than the training data, and report that in the first layer of the network common function words such as “and”, “a” and “the” go mostly to the generalists, while for the users trained on maths and programming text the domain-specific terms in the last layer go mostly to the specialists. Nothing here ran on a phone. The appendix states that every experiment except the centralised baseline ran on a single NVIDIA A100-SXM4-40GB, with four simulated users holding between roughly 400,000 and 1 million training tokens each, drawn from multilingual Wikipedia, SlimPajama, AG News and Common Corpus, and averaged over three seeds. The base models are GPT-2 124M and Llama 3.2 1B. Results in the paper’s table are mixed rather than uniform, with CoMiGS ahead of both baselines on multilingual Wikipedia, SlimPajama and Common Corpus, and behind federated averaging on AG News, at 33.53 against 31.84 with GPT-2 and 16.31 against 15.86 with Llama 3.2 1B, which is the one setup where the test data does not follow the training distribution. The second set of experiments gives users different numbers of adapters, two to four per device with one of them always a generalist, which is how the method is meant to cope with devices of unequal capacity. Against HetLoRA and FlexLoRA, two methods that vary the LoRA rank per device instead, the authors report lower perplexity on every in-distribution task, for example 18.68 against 21.41 and 24.63 on Common Corpus with Llama 3.2 1B. They put the overhead of their extra router at 1.25 percent more computation per forward pass and 1.25 percent more memory, with communication per round halved against federated averaging because only the generalist weights are sent. The paper names its own limits. The authors write that differential privacy on the aggregated generalist weights is something they “do not pursue here”, that robustness against attackers is beyond the scope of the work and that their aggregation carries no guarantee against adversarial parties, and that there is a limit to how much the generalists can prevent overfitting when the local task is easy. They also note that fine-tuning Llama 3.2 1B on multilingual Wikipedia or SlimPajama produced negligible improvements, which they attribute to overlap with its pre-training data, and that is why the Llama runs use Common Corpus instead. The code is on GitHub under the Apache 2.0 licence. Diagram: Fan et al., Figure 2 of the paper, licensed CC BY 4.0. The router and the specialists stay on the device. Source: https://arxiv.org/abs/2409.13931 Read the article: https://llmobile.news/ticker/on-device-collaborative-lm/Mixture of expertsResearchOpen sourceLlamahttps://llmobile.news/ticker/mobillama/https://llmobile.news/ticker/mobillama/Mon, 26 Feb 2024 19:59:00 +0100Researchers at MBZUAI published MobiLlama on February 26, 2024, a language model of 0.5B parameters aimed at devices that cannot host a large one. The stated aim is a fully transparent small model rather than open weights alone, so the authors release the complete pre-training data pipeline, the training and evaluation code, the weights and more than 300 intermediate checkpoints. They also derive a 0.8B version from the same design. The design point is a single feed-forward block shared by every transformer layer. Each transformer block normally carries its own feed-forward network, the part that reworks each token’s representation after attention, and the authors measure those layers as 65% of all trainable parameters in their 1.2B reference model. Reusing one block across all 22 layers cuts trainable parameters by 60%, from 1.2B to 0.52B, while keeping the 22 layers and the hidden size of 2048 that the reference model has. The authors report pre-training the shared version in 7 days and 26.6K A100 GPU hours, against 12 days and 46.1K hours for the unshared 1.2B model. Diagram: Thawakar et al. For deployment the authors measured a smartphone with a Snapdragon 685, a laptop with an i7 CPU and a desktop with an RTX 2080 Ti, using 4-bit GGUF builds on the phone and the laptop. On the phone they put MobiLlama 0.5B at 770 MB of RAM and 7.02 tok/s, drawing 5.32 mAh per 1,000 tokens at 13.02% CPU utilisation. They measured Llama 2 7B on the same phone at 4287 MB, 1.19 tok/s and 10.07 mAh per 1,000 tokens with 77.41% CPU utilisation, and Phi-2 2.7B at 1893 MB and 2.88 tok/s. On the laptop CPU the 0.5B model reaches 36.32 tok/s in 799 MB. MobiLlama 0.5B averages 46.00 across nine benchmarks from the Open LLM Leaderboard in the authors’ own evaluation, which they compare with 43.57 for pythia-410m, and the 0.8B version, made by widening the shared block, averages 46.67. The models were pre-trained on 1.2T tokens from the Amber dataset of LLM360, where the two largest parts are a curated web crawl and code. Weights and chat-tuned variants are on Hugging Face under Apache 2.0, and the repository carries the training code and an Android app package for running the model on a phone. Source: https://arxiv.org/abs/2402.16840 Read the article: https://llmobile.news/ticker/mobillama/ResearchOpen weightsOpen sourceAndroidQuantisationhttps://llmobile.news/ticker/tinyllava/https://llmobile.news/ticker/tinyllava/Thu, 22 Feb 2024 06:05:30 +0100Researchers at Beihang University and Tsinghua University posted TinyLLaVA on February 22, 2024, a framework for building small vision-language models out of an existing language model and an existing image encoder. Their best configuration, which the paper calls TinyLLaVA-3.1B, scores higher than the 7B LLaVA-1.5 on seven of the nine benchmarks in the paper’s comparison table, ties it on GQA at 62.0, and trails it on MME, where the same table gives LLaVA-1.5 1510.7 against 1464.9. The authors report 79.9 against 78.5 on VQAv2 and 75.8 against 63.4 on LLaVA-Bench-in-the-Wild, and put the model ahead of the 7B Qwen-VL on four of the five benchmarks they share, behind it on TextVQA at 59.1 against 63.8. The framework treats four things as variables, the small language model, the vision encoder that turns an image into patch features, the connector that maps those features into the language model’s embedding space, and the training recipe. The authors instantiate it with TinyLlama at 1.1B, StableLM-2 at 1.6B and Phi-2 at 2.7B as language models, CLIP at 0.3B and SigLIP at 0.4B as vision encoders, and either the 558k-caption LLaVA-1.5 data or the 1246k-caption ShareGPT4V data. Their base recipe updates only the connector during pre-training, while their share recipe freezes the first 12 layers of the vision encoder and updates the rest of the model. The ablations put most of the movement in the vision encoder and the recipe. Swapping CLIP for SigLIP raises the Phi-2 variant’s TextVQA score from 51.4 to 55.6 and the TinyLlama variant’s from 45.8 to 49.1, which the authors attribute in part to SigLIP running at 384 pixels against 336 and emitting 729 visual tokens against 576. A two-layer MLP connector beat a resampler on four of six benchmarks in the single comparison the paper runs, 74.0 against 70.1 on VQAv2 and 58.1 against 54.2 on GQA, with the resampler ahead on SQA-I and MM-Vet. The authors report that the share recipe, which fine-tunes part of the vision encoder rather than freezing all of it, improved every variant pre-trained on ShareGPT4V. Model size still helps when data and recipe are held fixed, and with CLIP under the base recipe the Phi-2 variant leads the StableLM-2 and TinyLlama ones on all seven benchmarks in the appendix table. The ordering flips once the recipe changes, with the 1.5B SigLIP TinyLlama model on the share recipe beating the 1.9B CLIP StableLM-2 model on the base recipe on six of seven benchmarks. The authors add a hallucination caveat, reporting that the share recipe lowers POPE accuracy for the StableLM-2 and Phi-2 variants even as their other scores rise, and they conjecture that smaller language models need more trainable parameters during pre-training while larger ones suffer from them. The paper carries no on-device measurement. It states no latency, no memory footprint and no target hardware, and argues the small-scale case from training and inference cost rather than from phone or edge deployment. The authors wrote that weights and code would be made public, and the project repository and the 3.1B weights carry the Apache 2.0 license. Source: https://arxiv.org/abs/2402.14289 Read the article: https://llmobile.news/ticker/tinyllava/ResearchBenchmarksPhiOpen weightsOpen sourcehttps://llmobile.news/ticker/tinyllama/https://llmobile.news/ticker/tinyllama/Thu, 04 Jan 2024 18:54:00 +0100Researchers from the StatNLP group at Singapore University of Technology and Design published TinyLlama on January 4, 2024, a language model of 1.1B parameters pretrained on 3 trillion tokens. The authors set out to see how a small model behaves when it is trained on far more tokens than the compute-optimal scaling law of Hoffmann et al. recommends for its size, and they state that this is, to their knowledge, the first attempt to train a model of around 1B parameters on that much data. The corpus mixes natural language from SlimPajama with code from the StarCoder training data at a sampling ratio of roughly 7 to 3, which the authors put at about 950 billion tokens once the two are merged and the GitHub part of SlimPajama is dropped to avoid duplication. TinyLlama passes over that set for approximately three epochs, which is how the 3 trillion cumulative figure is reached. Pretraining ran on 16 A100-40G GPUs, and the project repository states that the run started on September 1, 2023 and was budgeted at 90 days. TinyLlama copies the architecture and tokenizer of Meta’s Llama 2, so it carries rotary position embeddings, RMSNorm pre-normalisation and SwiGLU activations across 22 layers with a hidden size of 2048 and a 2048-token context. It also inherits grouped-query attention, where the 32 query heads share four groups of key and value representations instead of each head holding its own, which the authors use to cut memory bandwidth and speed up inference. They report a training throughput of 24,000 tokens per second per A100-40G GPU, and measure 3,456 GPU hours to cover 300 billion tokens against 4,830 hours for Pythia-1.0B and 7,920 for MPT-1.3B in their own comparison. They attribute that speed to existing open-source components rather than anything new, naming Fully Sharded Data Parallel for spreading training across nodes, FlashAttention-2, fused implementations of layer normalisation, the cross entropy loss and the rotary position embedding, and the fused SwiGLU module from xFormers. Across seven zero-shot commonsense reasoning benchmarks the authors’ own table puts TinyLlama at an average of 52.99, ahead of OPT-1.3B at 51.44, Pythia-1.4B at 51.33 and Pythia-1.0B at 48.30, with the widest margin on HellaSwag at 59.20 against 53.65 for OPT-1.3B. It does not lead every column, and Pythia-1.4B stays ahead on BoolQ at 63.27 against 57.83. On the InstructEval suite TinyLlama averages 19.87 against 17.72 for Pythia-1.4B and 16.95 for OPT-1.3B, with most of that difference coming from HumanEval, where it scores 9.15 against 4.27. The authors put the device case briefly, writing in the paper that the compact architecture “can enable end-user applications on mobile devices”, and the repository describes the model as suited to applications with a restricted computation and memory footprint. It puts the 4-bit quantised weights at 637 MB and names offline real-time machine translation and drafting tokens that a larger model then checks as the uses it has in mind. Neither the paper nor the repository reports a measurement on a phone. Weights and all intermediate checkpoints are on Hugging Face under Apache 2.0, with the pre-training code in the repository. Source: https://arxiv.org/abs/2401.02385 Read the article: https://llmobile.news/ticker/tinyllama/LlamaResearchOpen weightsOpen sourcehttps://llmobile.news/ticker/apple-mlx/https://llmobile.news/ticker/apple-mlx/Tue, 05 Dec 2023 01:04:00 +0100Apple machine learning research shipped the first public release of MLX on December 5, 2023, an array framework for machine learning on Apple silicon. The repository carries a v0.0.2 tag dated that day and the same version went up on PyPI minutes later, after an initial commit on November 28. Arrays in MLX live in shared memory, so an operation can run on the CPU or on the GPU without the data being copied from one place to the other. That behaviour comes from the hardware. Apple silicon uses a unified memory architecture in which the CPU and the GPU have direct access to the same memory pool, and the MLX documentation states that an array therefore has no device of its own. Code names the device when it runs an operation instead of moving an array to a device first, and when two operations on different devices depend on each other, the documentation says the MLX scheduler inserts the dependency between the streams automatically. Computation is lazy. The documentation states that operations only record a compute graph and that nothing is computed until an eval call asks for a result, which is what lets MLX apply function transformations such as automatic differentiation and automatic vectorisation to the graph. It gives memory as the second reason, since a model whose weights are created as float32 and then replaced with float16 before any evaluation peaks at half the memory eager computation would need. The MLX Swift bindings followed a week later, with an initial commit on December 12, 2023, and the package manifest lists macOS 14, iOS 17, tvOS 17 and visionOS 1 as supported platforms. The example apps build for iOS as well as macOS and include a chat client for language and vision-language models, a text generation demo that downloads weights from Hugging Face, and Stable Diffusion image generation. The LLMEval example uses the Increased Memory Limit entitlement on iOS, which its README attributes to the size of language model weights, and caps the MLX buffer cache at 20 MB. MLX LM is the Python package that runs and fine-tunes language models on top of MLX, first published to PyPI on January 12, 2024. It installs the mlx_lm.generate and mlx_lm.chat command line tools, whose default model is a 4-bit quantised Llama 3.2 3B Instruct from the mlx-community organisation on Hugging Face, and it can quantise and upload converted models back to the Hub. MLX is published under the MIT license, and the install documentation lists Apple silicon, macOS 14.0 or newer and Python 3.10 or newer for the macOS package, alongside CUDA and CPU-only builds for Linux added later. Source: https://github.com/ml-explore/mlx Read the article: https://llmobile.news/ticker/apple-mlx/AppleApple SiliconDeveloper toolsOpen sourceiOShttps://llmobile.news/ticker/llm-swift/https://llmobile.news/ticker/llm-swift/Mon, 04 Dec 2023 00:11:00 +0100The South Korean developer eastriverlee published the first release of LLM.swift late on December 3, 2023, a Swift package that wraps llama.cpp so an iOS or Mac app can load a GGUF model file and generate text on the device in a few lines. The launch example creates a model object from a file bundled with the app and awaits a reply that streams back token by token into a SwiftUI view. The package declared macOS 12, iOS 14, watchOS 4, tvOS 14 and visionOS 1 as its minimum versions, and required Swift 5.9. Screenshot: LLM.swift. The test app shipped in the first release README. Sitting on llama.cpp means the package reads GGUF files and nothing else, and the project states that any model llama.cpp runs should run here too, which is a different route from Hugging Face’s swift-transformers and its Core ML pipeline. Chat formatting was left to the app author at first, a closure that assembled the prompt plus a stop string that ends generation, and the release seven weeks later replaced that with ready-made templates for ChatML, Alpaca and Llama, with Mistral and Gemma added since. A developer adds the package through the Swift Package Manager by pointing at the repository, and the README recommends models of 3B parameters or fewer on mobile devices, reporting a Mistral 7B model running on an iPad Air 5th generation at 5-bit quantisation and on an iPhone 12 mini at 2-bit. The project has since put out 41 further releases, the most recent in July 2026, and version 3 made the chat template embedded in the GGUF file the default, added function calling and schema-constrained structured output, and switched to a prebuilt llama.cpp framework fetched from an upstream release by URL and checksum. The minimum versions have risen to iOS 16, macOS 13, watchOS 9, tvOS 16 and visionOS 1. Almost all of the roughly 150 commits are eastriverlee’s own, and the package is published under the MIT license. Source: https://github.com/eastriverlee/LLM.swift Read the article: https://llmobile.news/ticker/llm-swift/Developer toolsiOSllama.cppOpen sourceApple Siliconhttps://llmobile.news/ticker/tinychatengine/https://llmobile.news/ticker/tinychatengine/Thu, 24 Aug 2023 22:45:00 +0200Song Han’s group at MIT pushed the first commit of TinyChatEngine on August 24, 2023, according to the repository’s commit history. The library runs quantised language models in plain C and C++ with no dependency on bulky libraries, and the README names x86 CPUs from Intel and AMD, ARM devices including Apple M1 and M2 Macs and the Raspberry Pi, and Nvidia GPUs through CUDA. The lab’s own slides put LLaMA2-7B at 71 ms per token on an M1 Pro and 210 ms per token on an Intel i7-9750H laptop CPU. The compression behind it comes from two papers by the same lab. AWQ, for activation-aware weight quantisation, was submitted on June 1, 2023 and works from the finding that protecting only 1 percent of the weights, picked by looking at the activation distribution rather than the weights themselves, keeps most of the accuracy at 4 bits. SmoothQuant, submitted on November 18, 2022, handles 8-bit weights and 8-bit activations by shifting the hard part of the job from activations to weights through a mathematically equivalent transformation, and its authors report up to 1.56x speedup and 2x memory reduction. Diagram: MIT HAN Lab. The backend table lists W4A32, W4A8 and W8A8 on both x86 and ARM CPUs, W4A16 on Nvidia GPUs, and FP32 on CPU. TinyChatEngine reorders the 4-bit weights offline during model conversion rather than at runtime, in a layout tailored to each instruction set, with QM_ARM built for 128-bit SIMD and QM_x86 for 256-bit SIMD. The slides state that expanding the 4-bit weights to int8 and using int8 SIMD multiply-accumulate instead of FP32 gives 1.3x on the Intel i7-9750H and 3x on the M1 Pro. For the device comparison the lab measured LLaMA2-7B against PyTorch and llama.cpp. On the Intel i7-9750H it reports 210 ms per token for TinyChatEngine, 233 for llama.cpp and 1375 for PyTorch, which it labels a 6.5x gap. On an Nvidia Jetson Orin the figures are 108, 133 and 141 ms per token, and on the M1 Pro TinyChatEngine lands at 71 ms per token against 66 for llama.cpp, with PyTorch running out of memory. The lab publishes no Raspberry Pi timing. Chart: MIT HAN Lab's own figures, from the project slides. The model zoo ships LLaMA-7B, LLaMA2-7B-chat and LLaMA2-13B-chat plus OPT at 125M, 1.3B and 6.7B, in FP32, AWQ int4 and SmoothQuant int8 variants, with a download script that takes a per-device quantisation flag. macOS users install boost and llvm through Homebrew, and Xcode for the Metal compiler that enables Apple GPU support. The repository carries an MIT license, added the day after the first commit. For the Raspberry Pi the project recommends a board with 8 GB of RAM and says its testing ran on a Raspberry Pi 4 Model B Rev 1.4 with aarch64. Source: https://github.com/mit-han-lab/TinyChatEngine Read the article: https://llmobile.news/ticker/tinychatengine/QuantisationApple SiliconResearchOpen sourceDeveloper toolshttps://llmobile.news/ticker/hf-swift-transformers/https://llmobile.news/ticker/hf-swift-transformers/Tue, 08 Aug 2023 16:15:00 +0200Hugging Face published swift-transformers on August 8, 2023, the date of the repository’s first tag, a Swift package that gives iOS and Mac developers a transformers-like API for running language models on the device through Core ML. Pedro Cuenca introduced it as an in-development library built around four modules, Tokenizers, Hub, Generation and Models. The Tokenizers module ports the normalisation and pre-processing rules that the Python tokenizers library keeps as configuration files next to a model on the Hub, so an app does not hardcode those decisions per model and can swap one checkpoint for another. It covered BPE at release, the family used by the GPT models, Falcon and Llama, with Unigram and WordPiece listed as later work. The Hub module downloads those configuration files and the model metadata, and Generation implements greedy decoding and top-k sampling. The Models module wraps a converted Core ML package behind a LanguageModel type, and Core ML then decides whether to run it on the CPU, the GPU or the Neural Engine. Cuenca writes that flexible input shapes run on the CPU only, which is why Hugging Face’s exporters conversion tool switched to fixed sequence lengths, and that the Core ML path cached no attention keys and values at that point, both of which cap how long a context an app can work with. Hugging Face states it tested the package with Llama 2, Falcon, StarCoder and the GPT family including GPT2, distilgpt, GPT-NeoX and GPT-J, and that encoder-decoder models such as T5 and Flan were not supported. The release came alongside swift-chat, a Mac app that loads a Core ML model and doubles as a model tester, plus converted weights for Llama 2 7B and Falcon 7B on the Hub. The package is licensed under Apache 2.0. Screenshot: Hugging Face. The demo app exposes the generation parameters and reports throughput. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Video: Hugging Face. swift-chat generating a haiku from Llama 2 7B on an M1 MacBook Pro. Open the Hugging Face post Source: https://github.com/huggingface/swift-transformers Read the article: https://llmobile.news/ticker/hf-swift-transformers/Developer toolsiOSApple SiliconOpen sourceLlamahttps://llmobile.news/ticker/llmfarm/https://llmobile.news/ticker/llmfarm/Wed, 14 Jun 2023 18:14:00 +0200Artem Savkin published the first release of LLMFarm on June 14, 2023, an iOS and macOS app that loads a quantised model file kept on the device and holds a chat with it offline. The app is built on ggml and llama.cpp, and the README asks for iOS 16 or later and macOS 13 or later. Screenshot from the first release. Screenshot: LLMFarm. The app reads two llama.cpp file formats, the older ggjtv3 files with a .bin extension and GGUF files, according to the project FAQ. The first release covered LLaMA, GPT-NeoX and GPT-2 style models, with 3B builds of OpenLLaMa, StableLM and Dolly v2 among those it listed as working on an iPhone. The README has since added Gemma, Phi, Qwen, Mixtral, RWKV and LLaVA-style vision models, and marks most of them as running through Metal, Apple’s GPU interface, which it says does not work on Intel Macs. Quantisation happens before a model reaches the phone, and the project’s on-device notes list 7B Llama 2 and Orca builds at 3-bit and 4-bit as the ones tried on an iPhone 12 Pro Max, with no published token rates. Each chat writes a JSON file of inference options, which the first release documented with defaults of temperature 0.8, top-k 40, top-p 0.95, a 2048-token context and a repeat penalty of 1.1. The app also offers tail free sampling, locally typical sampling, Mirostat, greedy decoding and grammar-constrained output. Models arrive either through an add model button that imports a file from local storage or through a download list inside the app, and the sandboxed app keeps them in its own models directory. Memory sets the ceiling on an iPhone. The first release’s model table marked 3B models as working on an iPhone 12 Pro Max and 7B and 13B models as returning a “bad alloc” error, and the README said extended memory might be needed to solve it. The FAQ states that the Extended Virtual Addressing and increased-memory-limit entitlements are required to run models larger than 3B, and that the most common reason for a crash is lack of memory. LLMFarm is published under the MIT license, and the README now says the app is temporarily unavailable in TestFlight and the App Store. Source: https://github.com/guinmoon/LLMFarm Read the article: https://llmobile.news/ticker/llmfarm/llama.cppiOSiPhoneOpen sourceQuantisationhttps://llmobile.news/ticker/rwkv/https://llmobile.news/ticker/rwkv/Mon, 22 May 2023 15:57:41 +0200Bo Peng and 29 co-authors published RWKV on May 22, 2023, an architecture that trains in parallel like a transformer but runs as a recurrent network when it generates text. They released pretrained weights in six sizes from 169M to 14B parameters, all trained on the Pile, and describe the 14B model as by far the largest dense RNN trained to that point. A transformer keeps a cache of every token it has already processed, so both the memory it holds and the work it does for each new token grow as the context gets longer. RWKV carries a fixed-size state instead, which holds the same amount of data at token 10 and at token 10,000. The complexity table in the paper puts RWKV at O(Td) time and O(d) space for a sequence of T tokens, against O(T²d) time and O(T² + Td) space for a standard transformer, and the authors note that the arithmetic per token does not depend on context length at all. The authors measured text generation on an NVIDIA A100 with 80 GB in float32 and report cumulative time that rises in a straight line for RWKV while the transformer baselines curve upward. In their chart, RWKV-4 3B reaches about 10 seconds of cumulative GPU time after 1024 tokens, where OPT 2.7B, GPT-Neo 2.7B, BLOOM 3B and Pythia 2.8B need roughly 57 to 63 seconds. Chart: Peng et al., licensed CC BY 4.0. Cumulative time to generate 1024 tokens in float32 on an A100. On zero-shot benchmarks the paper puts RWKV close to transformers of comparable size trained on the same data. RWKV-4 at 169M scores 65.07 on PIQA where Pythia 160M scores 62.68 and GPT-Neo 125M scores 63.06, and at 14.2B it leads an interpolated GPT-level baseline on five of seven common sense tasks while trailing it on WinoGrande and ARC-Easy. The authors list the trade-off as a limitation, since funnelling everything through one fixed state limits how well the model recalls small details from very long contexts, and they report that RWKV is more sensitive to prompt wording than a transformer. Peng publishes the code as RWKV-LM under Apache 2.0, and the weights on Hugging Face carry the same license. The repository lists rwkv.cpp among community projects, a ggml-based implementation that runs 4-bit, 8-bit and float inference on CPU, while optimised iOS and Android inference sits in the README as a to-do rather than finished work. Peng states the goal in that list. RWKV is a RNN and very friendly for edge devices. Let’s make it possible to run a LLM on your phone. Source: https://arxiv.org/abs/2305.13048 Read the article: https://llmobile.news/ticker/rwkv/RWKVResearchOpen weightsMemoryOpen sourcehttps://llmobile.news/ticker/mlc-llm-consumer-devices/https://llmobile.news/ticker/mlc-llm-consumer-devices/Mon, 01 May 2023 18:00:00 +0200The MLC project released MLC LLM on May 1, 2023, a compiler and runtime that deploys language models natively on phones, browsers and consumer hardware rather than serving them from a data centre. The team pointed users to a TestFlight build to try the chat app on iOS. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Video: MLC. The iOS app generating a reply on the phone. Open the MLC post MLC LLM compiles models through the Apache TVM Unity stack and maps them to the Vulkan and Metal graphics APIs, which the team says covers the majority of consumer platforms. Targets named at launch include Windows, Linux, macOS and iOS, plus AMD, NVIDIA, Intel and Apple GPUs and a Steam Deck. A companion project, WebLLM, offloads the same models into web browsers through WebGPU. Three weeks later the team added RedPajama-3B across the same set of targets, with a browser demo and a command line tool for desktop machines. The code is on GitHub under the Apache 2.0 license. The project now exposes an OpenAI-compatible API through a REST server as well as Python, JavaScript, iOS and Android bindings, and covers Android GPUs via OpenCL alongside CUDA and ROCm on the desktop. Source: https://blog.mlc.ai/2023/05/01/bringing-accelerated-llm-to-consumer-hardware Read the article: https://llmobile.news/ticker/mlc-llm-consumer-devices/MLC LLMiOSiPhoneOpen sourceWebGPUhttps://llmobile.news/ticker/sherpa/https://llmobile.news/ticker/sherpa/Mon, 27 Mar 2023 20:46:00 +0200Bip-Rep published Sherpa on March 27, 2023, a Flutter app that runs llama.cpp on the phone itself, so the chat works with no network connection. The README asks users to bring their own weights, Meta’s 7B LLaMA converted to a ggml model file, which the app loads through a file picker, and version 2.1.0 added Vicuna support and macOS builds next to the Android and Windows ones. The project publishes no tokens per second figure and states no minimum memory, and the only device it names is a OnePlus 7 with 8 GB of RAM in the demo video below, which the README says was recorded at real speed. The code is MIT licensed. Development stopped three weeks after it started. All 43 commits fall between March 27 and April 14, 2023, the last of the seven releases went out on April 14, 2023, and the llama.cpp fork the app builds against was last touched on April 12, 2023, four months before llama.cpp moved to the GGUF format. Sherpa reads the older ggml files rather than the GGUF models that have been standard since, and requests for GGUF support have gone unanswered, with 23 of the 29 issues ever filed still open. A contributor answered one of them by writing that the original developers had abandoned the project, and pointed to his own fork, Maid. ▶A working chatGPT running on a mobile using llama.cppLoading connects your browser to www.youtube-nocookie.com, which may process your IP address and use cookies. Load contentOpen externallyVideo: tibzejoker. Source: https://github.com/Bip-Rep/sherpa Read the article: https://llmobile.news/ticker/sherpa/llama.cppAndroidLlamaOpen sourcehttps://llmobile.news/ticker/llama-cpp-first-release/https://llmobile.news/ticker/llama-cpp-first-release/Fri, 10 Mar 2023 20:50:00 +0100Georgi Gerganov pushed the first commits of llama.cpp on March 10, 2023, according to the repository’s commit history. The project is an inference engine for Meta’s LLaMA models written in plain C and C++, and the README states its goal as language model and vision-language model inference with minimal setup and state-of-the-art performance on a wide range of hardware, locally and in the cloud. The core implementation carries no external dependencies. It supports integer quantisation at 1.5, 2, 3, 4, 5, 6 and 8 bits, which is what brings models within reach of consumer memory budgets. Apple silicon is a first-class target through ARM NEON, the Accelerate framework and Metal. The project has since added backends for x86 with AVX, AVX2, AVX512 and AMX, RISC-V, NVIDIA CUDA, AMD HIP, Moore Threads MUSA, Vulkan, SYCL, OpenCL and WebGPU. It is published under the MIT license. Source: https://github.com/ggml-org/llama.cpp Read the article: https://llmobile.news/ticker/llama-cpp-first-release/llama.cppOpen sourceQuantisationApple SiliconDeveloper toolshttps://llmobile.news/ticker/tinygrad/https://llmobile.news/ticker/tinygrad/Sun, 18 Oct 2020 07:57:00 +0200George Hotz pushed the first commit of tinygrad on October 18, 2020, with the message “start tinygrad”, according to the repository’s commit history. The README describes an end-to-end deep learning stack with a tensor library, an autograd, a compiler that fuses and lowers kernels, and a JIT, positioned between PyTorch and karpathy/micrograd. It also states that an accelerator port needs a total of about 25 low level ops. tiny corp, the company Hotz founded, maintains it. The README lists backends for OpenCL, CPU, Metal, CUDA, AMD, NVIDIA, QCOM and WebGPU. The runtime documentation names the requirements for each one, Adreno 6xx series GPUs for the QCOM backend and M1 or newer Macs for Metal, with Metal 3.0 needed for bfloat support. The QCOM backend, which drives the Qualcomm GPU through the Adreno kernel driver interface instead of OpenCL, entered the repository on September 2, 2024, and the repository also carries a backend for Qualcomm’s Hexagon DSP in ops_dsp.py. tiny corp states on its site that tinygrad is used in openpilot to run the driving model on the Snapdragon 845 GPU, where it replaces Qualcomm’s SNPE. The company says its framework is faster than SNPE, loads ONNX files, supports training and allows attention, which SNPE does not because it only allows fixed weights. The repository’s test workflow compiles an openpilot model with an image pitch alignment set to match the Adreno 630, the GPU in the Snapdragon 845. The same workflow enforces the project’s size, failing the build when the repository goes over 26,500 lines as counted by sz.py. The repository ships a language model tool in tinygrad/llm that loads GGUF files, serves an OpenAI compatible API and benchmarks tokens per second, with a built-in model list that starts at quantised Llama 3.2 1B and Qwen3 0.6B. tinygrad is published under the MIT license, and the project recommends installing from source rather than from PyPI. Diagram: tinygrad. Source: https://github.com/tinygrad/tinygrad Read the article: https://llmobile.news/ticker/tinygrad/Open sourceDeveloper toolsQualcommApple SiliconWebGPU