https://llmobile.news/tags/apple-silicon/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.news/ticker/edge0-35b-iphone-demo/https://llmobile.news/ticker/edge0-35b-iphone-demo/Thu, 10 Sep 2026 16:08:02 +0200Samuel Zeng has released Edge0, an open-source framework for streaming mixture-of-experts inference, under Apache-2.0. In his launch post he reports a 35-billion-parameter model running on an iPhone at 1 to 2.5 GB of peak memory without a cloud connection. The repository describes three techniques. Expert weights are streamed from storage on demand, so peak memory is bounded by the active set. A trained prerouter predicts expert routing one step ahead so that loads overlap the forward pass, which the project credits with up to 59 percent higher throughput. Recover-LoRA pairs an int4-quantised frozen base with distilled LoRA adapters. Two tiers are published. edge0-35b has 40 layers and 256 experts, a checkpoint of roughly 23 GB and about 2.9 GiB of peak active memory. edge0-8b has 24 layers and 128 experts, a 4.2 GB checkpoint and about 1.0 GB peak. On a Mac mini M4 Pro with 24 GB, the project measures 14.9 to 17.7 tokens per second of decode for the 35B tier and 23.9 to 25.3 for the 8B tier, with prefill at 113 tokens per second cold and 140 warm for 35B. Quality loss against FP16 baselines is given as 3.9 points on average for the 35B tier and 2.8 points for the 8B tier. The repository states that the MLX backend runs on macOS with Apple Silicon, on M1 through M4, and requires Python 3.10 or newer. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Demo by Samuel Zeng. Watch original on X Source: https://x.com/SamuelZengML/status/2097861839287927139 Read the article: https://llmobile.news/ticker/edge0-35b-iphone-demo/iPhoneApple SiliconOpen weightsMixture of expertsMemoryhttps://llmobile.news/ticker/apertus-mini/https://llmobile.news/ticker/apertus-mini/Wed, 27 May 2026 23:40:00 +0200Researchers working in the Swiss AI Initiative published a technical report on May 27, 2026 describing three language models distilled from Apertus 8B, at 0.5B, 1.5B and 4B parameters. The authors are based at EPFL, ETH Zurich and ISTA, and the models sit on Hugging Face as the Apertus Mini collection. They inherit what the project calls a fully open and compliant release, meaning published weights, published data pipeline and published recipe, with pre-training restricted to openly available data that honours robots.txt opt-outs. The model card lists 1,811 natively supported languages. Each instruction-tuned model also ships in MLX format at 3, 4 and 6 bits, which the authors aimed at Apple devices and at memory-limited mobile and edge deployment. The 4-bit 4B weights are a 2.59 GB file against 7.65 GB for the same model in bfloat16, and the 1.5B and 0.5B come to 1.08 GB and 0.31 GB. The authors build the quantised copies with quantisation-aware distillation, a short recovery run that retrains the compressed weights against the 8B model’s own output distribution, and report 90% to 104% of the unquantised few-shot accuracy on ARC, HellaSwag, MMLU and WinoGrande. Separate FP8 and NVFP4 checkpoints target NVIDIA GPUs instead. The base model table averages ARC, HellaSwag, WinoGrande, XNLI, XCOPA and PIQA, and puts the three sizes at 51.79, 56.66 and 61.53 against 64.96 for the 8B teacher. The same table gives the Qwen3 base models 52.23, 57.51 and 62.14 at comparable sizes, so the distilled models land just under them on the average while leading on the two cross-lingual sets, 45.03 against 43.00 on XNLI and 63.82 against 61.82 on XCOPA at 4B. On a multilingual average of MMLU, TruthfulQA, ARC, instruction following and LogiQA for the instruction-tuned models, the authors report 0.318, 0.382 and 0.473 against 0.401, 0.457 and 0.521 for Qwen3 and 0.497 for Gemma 3 4B, and write that the family is weaker at instruction following and maths. Each model saw 1.7T tokens taken from the final and most heavily filtered phase of the original Apertus data, with no new sources added, packed into 4,096-token sequences and trained on 64 GH200 GPUs at the Swiss National Supercomputing Centre. The cost table puts the 4B run at 2.0E22 FLOPs and the single pass that produced the teacher’s logits for all three students at 1.4E22, against 3.7E23 FLOPs for the original Apertus 8B pre-training on 15T tokens, which the authors describe as making the whole family cost under 12% of the teacher. Swiss AI released 24 checkpoints in total, base, instruction-tuned and quantised, under the Apache 2.0 licence with an acceptable use policy attached, along with the training, post-training and quantisation code. Source: https://arxiv.org/abs/2605.29128 Read the article: https://llmobile.news/ticker/apertus-mini/Swiss AIDistillationQuantisationOpen weightsApple Siliconhttps://llmobile.news/ticker/apple-foundation-models-sdk/https://llmobile.news/ticker/apple-foundation-models-sdk/Wed, 25 Feb 2026 20:47:00 +0100Apple published python-apple-fm-sdk, Python bindings for its Foundation Models framework, with a first beta release on February 25, 2026 according to the repository’s release history. The package reaches the on-device model at the core of Apple Intelligence from Python on macOS, and installs with pip install apple-fm-sdk. The Python API follows the Swift one. SystemLanguageModel reports whether the model is available and, if not, why, while LanguageModelSession holds a conversation and its async respond call returns text or streams it. A @fm.generable decorator marks a Python class for the model to fill in, and fm.guide adds per-field constraints such as a numeric range, as the README shows with a Cat class whose age is bounded to 0 through 20. Tool calling and session transcripts are exposed as well. The bindings do not reimplement inference. The repository carries a Swift package named foundation-models-c that wraps the framework behind a C interface and builds it as a dynamic library, which the Python layer then calls. Apple writes in the documentation that the bindings run the Swift framework underneath, so evaluations reflect real on-device performance and behaviour, and its evaluation guide notes that inference calls are processed one at a time rather than in parallel at the macOS hardware level. Apple lists macOS 26.0 or later, Xcode 26.0 or later with its agreement accepted in the Xcode app, Python 3.10 or later, and Apple Intelligence switched on. PyPI carries only source distributions for the package, so the Swift component is compiled locally during installation, which is what the Xcode requirement covers. Apple Intelligence on the Mac runs on Apple silicon machines only. The code is under the Apache 2.0 license and the package metadata on PyPI classifies it as alpha. Apple pushed 0.1.0 to PyPI on March 8, 2026, added the Attachment API for sending images alongside text in 0.2.0 on June 8, 2026 for WWDC 2026, and exposed the model’s context size and the token count of a given input in 0.2.1 on June 29, 2026. The README states the project is not yet taking contributions. Source: https://github.com/apple/python-apple-fm-sdk Read the article: https://llmobile.news/ticker/apple-foundation-models-sdk/AppleDeveloper toolsApple SiliconOpen sourcehttps://llmobile.news/ticker/intelligence-per-watt/https://llmobile.news/ticker/intelligence-per-watt/Tue, 11 Nov 2025 07:33:00 +0100Researchers at Stanford University and Together AI profiled more than 20 local models on 1M real queries and report that at least one model with 20B or fewer active parameters answered 88.7% of them correctly. The best single model in their set, GPT-OSS-120B, reached 71.4% on its own, averaged across the four query collections. Their 2025 efficiency figure comes from that model running on an Apple M4 Max. The paper proposes intelligence per watt, mean task accuracy divided by the mean power the accelerator draws while answering, so a pairing that gets more answers right on the same power budget scores higher. The best local pairing each year moved from 7.92×10⁻⁴ accuracy per watt for Mixtral 8x7B on an NVIDIA Quadro RTX 6000 in 2023, to 1.80×10⁻³ for Llama 3.1 8B on an RTX 6000 Ada in 2024, to 4.18×10⁻³ for GPT-OSS-120B on an Apple M4 Max in 2025, which the authors put at a 5.3 times gain over two years. Query coverage for that single best pairing rose from 23.2% to 71.3% across the same three years. A companion metric divides accuracy by the energy a query consumes instead, so slow generation counts against a device. Diagram: Saad-Falcon et al., Figure 1 of the paper, licensed CC BY 4.0. The hardware panel separates the accelerators the authors treat as local from the cloud parts. The 1M queries come from four collections. The team took 500K prompts from WildChat, a month of real ChatGPT traffic, 500K from NaturalReasoning, 12K from MMLU Pro and 26.5K from SuperGPQA, all single-turn. WildChat has no ground truth, so GPT-4o judged whether a local model’s answer matched one written by Qwen3-235B, while the other three collections supplied reference answers. Every run used a batch size of 1, one query at a time. The hardware the authors call local is desktop and workstation class rather than mobile. Their 2025 numbers ran on an Apple Mac Studio with an M4 Max, 128 GB of unified memory and 480 W of continuous power, and the 2023 and 2024 points used NVIDIA workstation cards. No phone, tablet or wearable was profiled. Running the same Qwen3 models, the M4 Max reached 1.40 times lower accuracy per watt than an NVIDIA B200 and 1.6 to 2.3 times lower accuracy per joule, a gap the authors read as headroom for local accelerator design. For the savings case the team simulated 80.2M queries over 24 hours split between four small models on M4 Max machines and Qwen3-235B in the cloud. Perfect routing to the smallest capable model cut energy by 80.4% against sending everything to the frontier model, and a router that picks correctly 80% of the time cut it by 64.3%, with the perfect case described as a theoretical maximum. The authors note that coverage falls in technical domains such as architecture and engineering, that GPT-OSS-120B answered 88.9% of WildChat queries against 64.9% of NaturalReasoning ones, and that they release their profiling harness so the measurements can be repeated on new hardware. Source: https://arxiv.org/abs/2511.07885 Read the article: https://llmobile.news/ticker/intelligence-per-watt/ResearchBenchmarksApple SiliconNvidiaQwenhttps://llmobile.news/ticker/a19-pro-neural-accelerators/https://llmobile.news/ticker/a19-pro-neural-accelerators/Tue, 09 Sep 2025 20:00:00 +0200Apple announced the A19 Pro alongside the iPhone 17 Pro on September 9, 2025. Its 6-core GPU carries Neural Accelerators built into each core, which Apple says work together with the 16-core Neural Engine to power AI models, graphics and games. The company states that the chip enables running large local language models on the phone. The A19 Pro also has a 6-core CPU that Apple calls the fastest in any smartphone, along with a larger cache and more memory than the A18 Pro. Combined with a vapor chamber cooling system, Apple puts sustained performance at up to 40 percent above the previous generation. Image: Apple. The vapor chamber Apple credits for the sustained performance gain. Source: https://www.apple.com/newsroom/2025/09/apple-unveils-iphone-17-pro-and-iphone-17-pro-max/ Read the article: https://llmobile.news/ticker/a19-pro-neural-accelerators/AppleiPhoneChipsApple SiliconNPUhttps://llmobile.news/ticker/apple-foundation-models-report-2025/https://llmobile.news/ticker/apple-foundation-models-report-2025/Fri, 18 Jul 2025 01:37:00 +0200Apple posted its 2025 foundation models tech report to arXiv on July 17, 2025, giving the numbers behind the roughly 3-billion-parameter on-device model it shipped at WWDC25. The report’s Table 3 puts the price of compressing that model to 2 bits per weight at 3.4 points of MMLU, from 67.8 in 16 bits down to 64.4, with IFEval falling from 85.1 to 82.3. Apple reaches 2 bits through quantisation-aware training with a learnable scaling factor per weight tensor, and reports that a balanced set of four levels at -1.5, -0.5, 0.5 and 1.5 trains more smoothly than the usual -2, -1, 0 and 1. The embedding table is quantised to 4 bits and the key-value cache to 8 bits, and low-rank adapters are fine-tuned afterwards to claw back the quality the compression costs. The server model takes a different route, compressed after training to 3.56 bits per weight with Adaptive Scalable Texture Compression, a GPU texture format whose fixed-function decode hardware on Apple GPUs unpacks the weights at no cost to the compute cores. The report also spells out the KV cache sharing the framework announcement only named. Apple splits the transformer so that Block 1 holds 62.5 percent of the layers and Block 2 the remaining 37.5 percent with its key and value projections removed, reusing Block 1’s cache instead. That cuts KV cache memory by 37.5 percent, and because Block 2 produces no keys or values, prefill skips its computation entirely, which Apple says cuts time to first token by about the same 37.5 percent. Apple rebuilt how the on-device model is trained. The company first trained a dense model on about 14T tokens, sparse-upcycled it into a 64-expert mixture-of-experts version, where only part of the model runs per token, using 1T high-quality tokens, then retrained the dense model on its last 10 percent of tokens against a distillation loss from that MoE teacher. Apple states this cut the cost of training the teacher by 90 percent and removed the need for structural pruning. The server model was trained on 8192 v5p Cloud TPU chips for 13.4T tokens. For the language expansion the report states the models are designed to support 16 languages, built on a text tokenizer grown from a 100k to a 150k vocabulary and a continued pre-training mix whose multilingual share rose from 8 percent to 30 percent. In Apple’s own human evaluation on US English text prompts, graders preferred the on-device model over Qwen-2.5-3B on 35.3 percent of prompts against 12.9 percent the other way, while against Gemma-3-4B the split was 21.0 percent to 21.9 percent. The report is on arXiv under a Creative Commons Attribution 4.0 license. Chart: Apple's own human evaluation, from the tech report. Rows marked with an asterisk were tested against the compressed on-device model. Source: https://arxiv.org/abs/2507.13575 Read the article: https://llmobile.news/ticker/apple-foundation-models-report-2025/AppleApple SiliconQuantisationDistillationBenchmarkshttps://llmobile.news/ticker/locally-ai/https://llmobile.news/ticker/locally-ai/Sun, 13 Apr 2025 09:00:00 +0200Adrien Grondin released Locally AI on April 13, 2025, an iPhone and iPad app that downloads open-weight language models and runs them entirely on the device. The App Store listing gives that release date and lists the app as free with no in-app purchases. Once a model is on disk the app needs no account and no internet connection, and Grondin states that no data is collected and nothing is sent to a cloud service. The site at launch named Meta Llama 3.2 and Llama 3.1, Google Gemma 2, Qwen 2 and DeepSeek R1 as the models on offer, and described them as language and vision models, so a picture can go into the prompt alongside text. A user picks a model, downloads it once and can edit the system prompt the model runs under. Grondin builds the app on MLX, Apple’s machine learning framework for Apple silicon, and attributes the performance to unified memory, where the CPU and the GPU reach the same memory pool so arrays need not be copied between them. The app fetches open-weight model files itself rather than calling any model that ships with the operating system. Neither the site nor the App Store entry names a minimum chip. The site says only that Locally AI covers recent iPhone and iPad models and is optimised for Apple silicon, and the listing asks for iOS 18.1 or iPadOS 18.1 or later. No generation speed, memory requirement or model file size is given anywhere in the developer’s own material, and the launch site was a bought template whose team, testimonial and FAQ sections still carried placeholder Latin text. Update, April 8, 2026. Grondin added support for Apple’s on-device Foundation Models framework in September 2025, so the app can open a conversation with the model built into iOS 26 without downloading anything, which Simon Willison recorded on September 21, 2025. A Mac version followed and requires macOS 26 or later. LM Studio announced on April 8, 2026 that Locally AI had joined the company and that Grondin had moved with it, and the App Store entry now reads Locally AI by LM Studio with Element Labs Inc as the seller. Source: https://locallyai.app/ Read the article: https://llmobile.news/ticker/locally-ai/AppleApple SiliconMLXiOSiPhonehttps://llmobile.news/ticker/huggingsnap/https://llmobile.news/ticker/huggingsnap/Wed, 19 Mar 2025 16:16:00 +0100Hugging Face published HuggingSnap, an iPhone app that describes what the camera is pointed at, with the model running on the phone rather than in the cloud. The app’s runtime configuration loads SmolVLM2 at 500M parameters in Apple’s MLX format. Hugging Face had trailed the app in its SmolVLM2 announcement on February 20, 2025, writing that the 500M model lets users analyse and understand video content directly on their device with no cloud required. The README describes the app as a way to point the camera and have text translated or summarised, identify plants and animals and learn more about surrounding objects. Inference runs through MLX Swift, Apple’s machine learning framework for Apple silicon, with a modified fork of Apple’s MLX example code supplying the vision-language support. The app pulls the weights from the Hugging Face Hub the first time it runs, then holds Metal memory to 82 percent of what iOS reports as available to the process. Input comes from the live camera, from stills and clips captured in the app, and from photos and videos picked out of the library. A tap on the shutter takes a picture and a press and hold records video, after which the app offers a describe action for images and a summarise action for video, with questions typed or dictated. The app’s own interface states that photos and videos are processed entirely on the device and that no data is sent to the cloud. The README states that installation needs an iPhone running iOS 18, and the Xcode project sets iOS 18.0 as the deployment target and iPhone as the only device family, with no Mac Catalyst or visionOS build. TechCrunch reported the app as a free App Store download on March 19, 2025, quoting the listing as saying it works offline and processes all data on the phone. The SmolVLM2 weights are Apache 2.0, while the app repository itself carries no licence file. Source: https://github.com/huggingface/HuggingSnap Read the article: https://llmobile.news/ticker/huggingsnap/Hugging FaceiOSiPhoneApple SiliconOpen weightshttps://llmobile.news/ticker/flower-intelligence/https://llmobile.news/ticker/flower-intelligence/Wed, 12 Mar 2025 10:00:00 +0100Flower Labs released Flower Intelligence on March 12, 2025 as a preview, an open source inference library that runs a language model on the user’s own device and only moves the work to a remote service if the app and the user allow it. The company shipped SDKs for Swift and TypeScript, and the TypeScript build defaults to Llama 3.2 1B in 16-bit precision. The library picks a local engine from the environment it finds itself in. The TypeScript package depends on MLC’s WebLLM, which executes models in the browser through WebGPU, and on Hugging Face transformers.js, which it uses when the code runs under Node.js rather than in a page. The Swift package is built on MLX Swift and Hugging Face swift-transformers, and declares iOS 16 and macOS 14 as its minimum platforms, so local inference on Apple hardware goes through MLX. Flower Labs’ model reference covers Llama 3.2 at 1B and 3B, Llama 3.1 8B, SmolLM2 at 135M, 360M and 1.7B, and DeepSeek R1 Distill Llama 8B, most of them as 4-bit builds. Which of those run on device depends on the backend. Node.js is limited to the 1B and SmolLM2 entries, while the 8B models run only in the browser and on the MLX Swift path, and Llama 3.2 3B in 16-bit is MLX Swift only. Remote handoff is off by default. An application has to set the remoteHandoff property and supply an API key before the library will contact Flower Confidential Remote Compute, and even with both in place a chat call tries a local engine first and retries remotely only when no local engine matches the model or the local run fails. Individual requests can be pinned with forceLocal or forceRemote. Flower Labs describes the remote service as a private extension of the device that costs the developer no extra work, and named Mozilla Thunderbird as an early adopter building its Thunderbird Assist feature on the library. Ryan Sipes, Managing Director for Mozilla Thunderbird, tied that choice to the mail client’s user base in the announcement. Our 20 million users expect data privacy from every feature we build. Both SDKs are published under the Apache 2.0 license, the TypeScript one as @flwr/flwr on npm and the Swift one as a package in the Flower repository. Flower Labs called the release inference-only and listed fine-tuning, RAG and pre-training as later steps, with the remote compute service available on application for early access. Source: https://flower.ai/blog/2025-03-12-flower-intelligence Read the article: https://llmobile.news/ticker/flower-intelligence/Open sourceDeveloper toolsLlamaWebGPUApple Siliconhttps://llmobile.news/ticker/exolabs-benchmarks/https://llmobile.news/ticker/exolabs-benchmarks/Wed, 25 Dec 2024 19:55:00 +0100EXO Labs opened a public benchmark site on December 25, 2024, as the first entry in a twelve-part series, publishing generation speed and time to first token measured on real consumer devices. EXO states these are measurements rather than estimates. In the data behind the site, an iPhone 15 Pro with 8 GB of RAM generates 14.0 tok/s on Llama 3.1 8B and takes 1.67 seconds to return its first token. A Galaxy S24 Ultra with 12 GB manages 5.6 tok/s on the same 8B model, with 4.16 seconds to the first token, according to EXO’s published results. On the smaller Llama 3.2 1B the iPhone reaches 56.1 tok/s and the Galaxy 22.5 tok/s. A single Mac mini M4 Pro with 24 GB sits at 49.3 tok/s on Llama 3.1 8B, which puts the phone at under a third of the Mac’s rate on the same model. EXO splits a model into contiguous slices of its layers and hands each slice to a different device, so a model too large for one machine can run across several. The activations passed from one device to the next are small, under 4 KB for Llama 3.2 3B, and EXO says the bottleneck is network latency rather than bandwidth. Pooling phones does not speed up a single reply. A cluster of two iPhone 15 Pros and one Galaxy S24 Ultra runs Llama 3.1 8B at 7.0 tok/s, below the 14.0 tok/s of one iPhone alone, which EXO attributes to network overhead when the model already fits on a single device. Three Mac mini M4 Pros run Llama 3.3 70B at 4.1 tok/s, a model that does not fit in the 24 GB of any one of them. Every entry carries a timestamp, a git commit and the identifier of the automated run that produced it, and the site is free to browse. EXO says it is expanding the suite towards mixed device types and different quantisations, and the exo runtime itself is on GitHub under Apache 2.0. Source: https://blog.exolabs.net/day-1/ Read the article: https://llmobile.news/ticker/exolabs-benchmarks/BenchmarksiPhoneAndroidLlamaApple Siliconhttps://llmobile.news/ticker/apple-intelligence-foundation-models/https://llmobile.news/ticker/apple-intelligence-foundation-models/Mon, 10 Jun 2024 21:00:00 +0200Apple introduced Apple Intelligence on June 10, 2024, built on a roughly 3-billion-parameter language model that runs on the device and a larger model that runs on Apple silicon servers through Private Cloud Compute. The on-device model uses a 49K token vocabulary, the server model 100K. Apple compresses the on-device model with a mix of 2-bit and 4-bit weights, averaging 3.7 bits per weight, and says an option at 3.5 bits per weight costs no significant quality. On an iPhone 15 Pro the company reports a time to first token of about 0.6 milliseconds per prompt token and 30 tokens per second, both measured before token speculation. Rather than ship a separate model per feature, Apple swaps LoRA adapters on top of the shared base model. The adapters are stored in 16 bits and, at rank 16, take tens of megabytes each. Apple says human graders preferred its models over comparable ones including GPT-3.5, GPT-4, Llama, Mistral and Gemma across its evaluations. Diagram: Apple. Source: https://machinelearning.apple.com/research/introducing-apple-foundation-models Read the article: https://llmobile.news/ticker/apple-intelligence-foundation-models/AppleiPhoneiOSQuantisationApple Siliconhttps://llmobile.news/ticker/openelm/https://llmobile.news/ticker/openelm/Tue, 23 Apr 2024 01:12:00 +0200Apple posted OpenELM to arXiv on April 22, 2024, a family of four language models at 270M, 450M, 1.1B and 3B parameters, each also released in an instruction-tuned version. The name stands for Open-source Efficient Language Models, and the efficiency comes from layer-wise scaling, which spreads the parameter budget unevenly across the transformer’s layers instead of giving every layer the same shape. Table 1 of the paper gives the 1.1B model an average of 45.93% across the OpenLLM leaderboard tasks against 43.57% for OLMo 1.2B, a gap the authors state as 2.36%, reached on 1.5T pre-training tokens against OLMo’s 3.0T. Layer-wise scaling varies the number of attention heads and the width of the feed-forward network from layer to layer, keeping layers near the input narrower and widening them towards the output, so a layer late in the stack holds more parameters than an early one. Apple trained the four variants for 350k iterations with its CoreNet library on public data only, drawing on RefinedWeb, a deduplicated PILE, a subset of RedPajama and a subset of Dolma v1.6, about 1.8T tokens in total. Text is filtered and tokenised on the fly rather than pre-tokenised, using the same tokeniser as Llama. The paper benchmarks the models on an Apple MacBook Pro with an M2 Max chip and 64 GiB of RAM running macOS 14.4.1, through a port of OpenELM to MLX v0.10.0, and separately on a Linux workstation with an Intel i9-13900KF and an NVIDIA RTX 4090. On the MacBook in BFloat16, Apple measures the 270M model generating 212.40 tok/s and the 3B model 33.96 tok/s, with the 4-bit versions at 256.35 and 60.33 tok/s. On the CUDA machine OpenELM 1.08B is slower than OLMo 1.18B, at 92.15 tok/s against 203.40, which the authors attribute to a naive RMSNorm implementation and to OpenELM having 113 normalisation layers where OLMo has 33. Apple presents the release as the complete framework rather than weights and inference code, covering the training and evaluation code for public datasets, the training logs, multiple checkpoints, the pre-training configurations and the code that converts a model to MLX for inference and fine-tuning on Apple devices, with pre-converted bf16 and 4-bit checkpoints for all four sizes. The eight models are on Hugging Face and the code is in CoreNet. Both the model cards and CoreNet name the terms as the Apple sample code license, which is Apple’s own license and not one of the standard open source licenses. Source: https://arxiv.org/abs/2404.14619 Read the article: https://llmobile.news/ticker/openelm/AppleApple SiliconResearchOpen weightsBenchmarkshttps://llmobile.news/ticker/stable-lm-2-zephyr/https://llmobile.news/ticker/stable-lm-2-zephyr/Tue, 27 Feb 2024 20:00:00 +0100Stability AI published the Stable LM 2 1.6B technical report on February 27, 2024, describing a decoder-only model of 1.6B parameters trained from scratch on roughly 2 trillion tokens of public data. The mix is multilingual by design, with web text in German, Spanish, French, Italian, Dutch and Portuguese alongside English, and the report states the finished model handles all seven. The model has 24 layers and a context length of 4096 tokens, which the authors list as one of the things they want to extend. In the report’s multilingual table, which averages translated versions of four English benchmarks and a next-word prediction set across those seven languages, the base model scores 40.5, above Google’s Gemma 2B at 39.8 with 2.5B parameters and Microsoft’s Phi-2 at 34.6 with 2.7B. The chat variant reaches 41.5 and Stability’s own 3B model 41.7. The authors state they used no multilingual data during fine-tuning, so the language coverage comes out of pre-training alone. On the six English tasks of the Hugging Face Open LLM Leaderboard, the same report’s table puts the base model at an average of 45.3, behind Alibaba’s Qwen 1.5 1.8B at 46.6 and Gemma 2B at 46.5. The instruction-tuned StableLM 2 Zephyr 1.6B averages 49.7, the highest of the sub-2B entries in that table, ahead of Phi-1.5 at 47.7 and behind the 2.7B Phi-2 at 61.3. The report’s abstract calls Stable LM 2 1.6B the state-of-the-art open model under 2B parameters at the time of publication. Stability built the Zephyr variant in three stages. It fine-tuned the base model on seven public instruction datasets, 826,938 samples after dropping conversations longer than eight turns, then aligned the result with direct preference optimisation, which trains on pairs of preferred and rejected answers, using UltraFeedback and Intel’s Orca Pairs and borrowing most hyperparameters from the recipe behind Hugging Face’s Zephyr 7B. A third stage taught the model who made it and what a language model cannot do, using 10,000 opening messages the base model wrote for itself. The variant scores 5.42 on MT-Bench, where a larger model judges answers to open-ended questions, above Qwen 1.5 1.8B Chat at 5.29 and Gemma 2B Instruct at 5.19 and below Stability’s own StableLM Zephyr 3B at 6.64. Chart: Bellagente et al., the authors' own figures. Stability put both sets of weights on Hugging Face along with quantised builds for llama.cpp, Apple’s MLX and Intel’s OpenVINO. The report measures 127 tok/s at 11 W for the 4-bit MLX build on an M2 Mac mini with 8 GB, against 71 tok/s at 6 W in FP16, and 99 tok/s for a 4-bit GGUF build on a 2023 MacBook Pro with 16 GB, while stating these figures are meant as a practical reference rather than rigorous benchmarking. Neither release is open source. The report says the model ships under a non-commercial licence, the Zephyr model card names the Stability AI Non-Commercial Research Community License and points commercial users to the company, and the base model card now carries the Stability AI Community License with commercial use running through a Stability membership. Source: https://arxiv.org/abs/2402.17834 Read the article: https://llmobile.news/ticker/stable-lm-2-zephyr/Stability AIResearchBenchmarksOpen weightsApple Siliconhttps://llmobile.news/ticker/apple-mlx/https://llmobile.news/ticker/apple-mlx/Tue, 05 Dec 2023 01:04:00 +0100Apple machine learning research shipped the first public release of MLX on December 5, 2023, an array framework for machine learning on Apple silicon. The repository carries a v0.0.2 tag dated that day and the same version went up on PyPI minutes later, after an initial commit on November 28. Arrays in MLX live in shared memory, so an operation can run on the CPU or on the GPU without the data being copied from one place to the other. That behaviour comes from the hardware. Apple silicon uses a unified memory architecture in which the CPU and the GPU have direct access to the same memory pool, and the MLX documentation states that an array therefore has no device of its own. Code names the device when it runs an operation instead of moving an array to a device first, and when two operations on different devices depend on each other, the documentation says the MLX scheduler inserts the dependency between the streams automatically. Computation is lazy. The documentation states that operations only record a compute graph and that nothing is computed until an eval call asks for a result, which is what lets MLX apply function transformations such as automatic differentiation and automatic vectorisation to the graph. It gives memory as the second reason, since a model whose weights are created as float32 and then replaced with float16 before any evaluation peaks at half the memory eager computation would need. The MLX Swift bindings followed a week later, with an initial commit on December 12, 2023, and the package manifest lists macOS 14, iOS 17, tvOS 17 and visionOS 1 as supported platforms. The example apps build for iOS as well as macOS and include a chat client for language and vision-language models, a text generation demo that downloads weights from Hugging Face, and Stable Diffusion image generation. The LLMEval example uses the Increased Memory Limit entitlement on iOS, which its README attributes to the size of language model weights, and caps the MLX buffer cache at 20 MB. MLX LM is the Python package that runs and fine-tunes language models on top of MLX, first published to PyPI on January 12, 2024. It installs the mlx_lm.generate and mlx_lm.chat command line tools, whose default model is a 4-bit quantised Llama 3.2 3B Instruct from the mlx-community organisation on Hugging Face, and it can quantise and upload converted models back to the Hub. MLX is published under the MIT license, and the install documentation lists Apple silicon, macOS 14.0 or newer and Python 3.10 or newer for the macOS package, alongside CUDA and CPU-only builds for Linux added later. Source: https://github.com/ml-explore/mlx Read the article: https://llmobile.news/ticker/apple-mlx/AppleApple SiliconDeveloper toolsOpen sourceiOShttps://llmobile.news/ticker/llm-swift/https://llmobile.news/ticker/llm-swift/Mon, 04 Dec 2023 00:11:00 +0100The South Korean developer eastriverlee published the first release of LLM.swift late on December 3, 2023, a Swift package that wraps llama.cpp so an iOS or Mac app can load a GGUF model file and generate text on the device in a few lines. The launch example creates a model object from a file bundled with the app and awaits a reply that streams back token by token into a SwiftUI view. The package declared macOS 12, iOS 14, watchOS 4, tvOS 14 and visionOS 1 as its minimum versions, and required Swift 5.9. Screenshot: LLM.swift. The test app shipped in the first release README. Sitting on llama.cpp means the package reads GGUF files and nothing else, and the project states that any model llama.cpp runs should run here too, which is a different route from Hugging Face’s swift-transformers and its Core ML pipeline. Chat formatting was left to the app author at first, a closure that assembled the prompt plus a stop string that ends generation, and the release seven weeks later replaced that with ready-made templates for ChatML, Alpaca and Llama, with Mistral and Gemma added since. A developer adds the package through the Swift Package Manager by pointing at the repository, and the README recommends models of 3B parameters or fewer on mobile devices, reporting a Mistral 7B model running on an iPad Air 5th generation at 5-bit quantisation and on an iPhone 12 mini at 2-bit. The project has since put out 41 further releases, the most recent in July 2026, and version 3 made the chat template embedded in the GGUF file the default, added function calling and schema-constrained structured output, and switched to a prebuilt llama.cpp framework fetched from an upstream release by URL and checksum. The minimum versions have risen to iOS 16, macOS 13, watchOS 9, tvOS 16 and visionOS 1. Almost all of the roughly 150 commits are eastriverlee’s own, and the package is published under the MIT license. Source: https://github.com/eastriverlee/LLM.swift Read the article: https://llmobile.news/ticker/llm-swift/Developer toolsiOSllama.cppOpen sourceApple Siliconhttps://llmobile.news/ticker/tinychatengine/https://llmobile.news/ticker/tinychatengine/Thu, 24 Aug 2023 22:45:00 +0200Song Han’s group at MIT pushed the first commit of TinyChatEngine on August 24, 2023, according to the repository’s commit history. The library runs quantised language models in plain C and C++ with no dependency on bulky libraries, and the README names x86 CPUs from Intel and AMD, ARM devices including Apple M1 and M2 Macs and the Raspberry Pi, and Nvidia GPUs through CUDA. The lab’s own slides put LLaMA2-7B at 71 ms per token on an M1 Pro and 210 ms per token on an Intel i7-9750H laptop CPU. The compression behind it comes from two papers by the same lab. AWQ, for activation-aware weight quantisation, was submitted on June 1, 2023 and works from the finding that protecting only 1 percent of the weights, picked by looking at the activation distribution rather than the weights themselves, keeps most of the accuracy at 4 bits. SmoothQuant, submitted on November 18, 2022, handles 8-bit weights and 8-bit activations by shifting the hard part of the job from activations to weights through a mathematically equivalent transformation, and its authors report up to 1.56x speedup and 2x memory reduction. Diagram: MIT HAN Lab. The backend table lists W4A32, W4A8 and W8A8 on both x86 and ARM CPUs, W4A16 on Nvidia GPUs, and FP32 on CPU. TinyChatEngine reorders the 4-bit weights offline during model conversion rather than at runtime, in a layout tailored to each instruction set, with QM_ARM built for 128-bit SIMD and QM_x86 for 256-bit SIMD. The slides state that expanding the 4-bit weights to int8 and using int8 SIMD multiply-accumulate instead of FP32 gives 1.3x on the Intel i7-9750H and 3x on the M1 Pro. For the device comparison the lab measured LLaMA2-7B against PyTorch and llama.cpp. On the Intel i7-9750H it reports 210 ms per token for TinyChatEngine, 233 for llama.cpp and 1375 for PyTorch, which it labels a 6.5x gap. On an Nvidia Jetson Orin the figures are 108, 133 and 141 ms per token, and on the M1 Pro TinyChatEngine lands at 71 ms per token against 66 for llama.cpp, with PyTorch running out of memory. The lab publishes no Raspberry Pi timing. Chart: MIT HAN Lab's own figures, from the project slides. The model zoo ships LLaMA-7B, LLaMA2-7B-chat and LLaMA2-13B-chat plus OPT at 125M, 1.3B and 6.7B, in FP32, AWQ int4 and SmoothQuant int8 variants, with a download script that takes a per-device quantisation flag. macOS users install boost and llvm through Homebrew, and Xcode for the Metal compiler that enables Apple GPU support. The repository carries an MIT license, added the day after the first commit. For the Raspberry Pi the project recommends a board with 8 GB of RAM and says its testing ran on a Raspberry Pi 4 Model B Rev 1.4 with aarch64. Source: https://github.com/mit-han-lab/TinyChatEngine Read the article: https://llmobile.news/ticker/tinychatengine/QuantisationApple SiliconResearchOpen sourceDeveloper toolshttps://llmobile.news/ticker/hf-swift-transformers/https://llmobile.news/ticker/hf-swift-transformers/Tue, 08 Aug 2023 16:15:00 +0200Hugging Face published swift-transformers on August 8, 2023, the date of the repository’s first tag, a Swift package that gives iOS and Mac developers a transformers-like API for running language models on the device through Core ML. Pedro Cuenca introduced it as an in-development library built around four modules, Tokenizers, Hub, Generation and Models. The Tokenizers module ports the normalisation and pre-processing rules that the Python tokenizers library keeps as configuration files next to a model on the Hub, so an app does not hardcode those decisions per model and can swap one checkpoint for another. It covered BPE at release, the family used by the GPT models, Falcon and Llama, with Unigram and WordPiece listed as later work. The Hub module downloads those configuration files and the model metadata, and Generation implements greedy decoding and top-k sampling. The Models module wraps a converted Core ML package behind a LanguageModel type, and Core ML then decides whether to run it on the CPU, the GPU or the Neural Engine. Cuenca writes that flexible input shapes run on the CPU only, which is why Hugging Face’s exporters conversion tool switched to fixed sequence lengths, and that the Core ML path cached no attention keys and values at that point, both of which cap how long a context an app can work with. Hugging Face states it tested the package with Llama 2, Falcon, StarCoder and the GPT family including GPT2, distilgpt, GPT-NeoX and GPT-J, and that encoder-decoder models such as T5 and Flan were not supported. The release came alongside swift-chat, a Mac app that loads a Core ML model and doubles as a model tester, plus converted weights for Llama 2 7B and Falcon 7B on the Hub. The package is licensed under Apache 2.0. Screenshot: Hugging Face. The demo app exposes the generation parameters and reports throughput. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Video: Hugging Face. swift-chat generating a haiku from Llama 2 7B on an M1 MacBook Pro. Open the Hugging Face post Source: https://github.com/huggingface/swift-transformers Read the article: https://llmobile.news/ticker/hf-swift-transformers/Developer toolsiOSApple SiliconOpen sourceLlamahttps://llmobile.news/ticker/llama-cpp-first-release/https://llmobile.news/ticker/llama-cpp-first-release/Fri, 10 Mar 2023 20:50:00 +0100Georgi Gerganov pushed the first commits of llama.cpp on March 10, 2023, according to the repository’s commit history. The project is an inference engine for Meta’s LLaMA models written in plain C and C++, and the README states its goal as language model and vision-language model inference with minimal setup and state-of-the-art performance on a wide range of hardware, locally and in the cloud. The core implementation carries no external dependencies. It supports integer quantisation at 1.5, 2, 3, 4, 5, 6 and 8 bits, which is what brings models within reach of consumer memory budgets. Apple silicon is a first-class target through ARM NEON, the Accelerate framework and Metal. The project has since added backends for x86 with AVX, AVX2, AVX512 and AMX, RISC-V, NVIDIA CUDA, AMD HIP, Moore Threads MUSA, Vulkan, SYCL, OpenCL and WebGPU. It is published under the MIT license. Source: https://github.com/ggml-org/llama.cpp Read the article: https://llmobile.news/ticker/llama-cpp-first-release/llama.cppOpen sourceQuantisationApple SiliconDeveloper toolshttps://llmobile.news/ticker/tinygrad/https://llmobile.news/ticker/tinygrad/Sun, 18 Oct 2020 07:57:00 +0200George Hotz pushed the first commit of tinygrad on October 18, 2020, with the message “start tinygrad”, according to the repository’s commit history. The README describes an end-to-end deep learning stack with a tensor library, an autograd, a compiler that fuses and lowers kernels, and a JIT, positioned between PyTorch and karpathy/micrograd. It also states that an accelerator port needs a total of about 25 low level ops. tiny corp, the company Hotz founded, maintains it. The README lists backends for OpenCL, CPU, Metal, CUDA, AMD, NVIDIA, QCOM and WebGPU. The runtime documentation names the requirements for each one, Adreno 6xx series GPUs for the QCOM backend and M1 or newer Macs for Metal, with Metal 3.0 needed for bfloat support. The QCOM backend, which drives the Qualcomm GPU through the Adreno kernel driver interface instead of OpenCL, entered the repository on September 2, 2024, and the repository also carries a backend for Qualcomm’s Hexagon DSP in ops_dsp.py. tiny corp states on its site that tinygrad is used in openpilot to run the driving model on the Snapdragon 845 GPU, where it replaces Qualcomm’s SNPE. The company says its framework is faster than SNPE, loads ONNX files, supports training and allows attention, which SNPE does not because it only allows fixed weights. The repository’s test workflow compiles an openpilot model with an image pitch alignment set to match the Adreno 630, the GPU in the Snapdragon 845. The same workflow enforces the project’s size, failing the build when the repository goes over 26,500 lines as counted by sz.py. The repository ships a language model tool in tinygrad/llm that loads GGUF files, serves an OpenAI compatible API and benchmarks tokens per second, with a built-in model list that starts at quantised Llama 3.2 1B and Qwen3 0.6B. tinygrad is published under the MIT license, and the project recommends installing from source rather than from PyPI. Diagram: tinygrad. Source: https://github.com/tinygrad/tinygrad Read the article: https://llmobile.news/ticker/tinygrad/Open sourceDeveloper toolsQualcommApple SiliconWebGPU