https://llmobile.news/tags/iphone/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.news/ticker/edge0-35b-iphone-demo/https://llmobile.news/ticker/edge0-35b-iphone-demo/Thu, 10 Sep 2026 16:08:02 +0200Samuel Zeng has released Edge0, an open-source framework for streaming mixture-of-experts inference, under Apache-2.0. In his launch post he reports a 35-billion-parameter model running on an iPhone at 1 to 2.5 GB of peak memory without a cloud connection. The repository describes three techniques. Expert weights are streamed from storage on demand, so peak memory is bounded by the active set. A trained prerouter predicts expert routing one step ahead so that loads overlap the forward pass, which the project credits with up to 59 percent higher throughput. Recover-LoRA pairs an int4-quantised frozen base with distilled LoRA adapters. Two tiers are published. edge0-35b has 40 layers and 256 experts, a checkpoint of roughly 23 GB and about 2.9 GiB of peak active memory. edge0-8b has 24 layers and 128 experts, a 4.2 GB checkpoint and about 1.0 GB peak. On a Mac mini M4 Pro with 24 GB, the project measures 14.9 to 17.7 tokens per second of decode for the 35B tier and 23.9 to 25.3 for the 8B tier, with prefill at 113 tokens per second cold and 140 warm for 35B. Quality loss against FP16 baselines is given as 3.9 points on average for the 35B tier and 2.8 points for the 8B tier. The repository states that the MLX backend runs on macOS with Apple Silicon, on M1 through M4, and requires Python 3.10 or newer. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Demo by Samuel Zeng. Watch original on X Source: https://x.com/SamuelZengML/status/2097861839287927139 Read the article: https://llmobile.news/ticker/edge0-35b-iphone-demo/iPhoneApple SiliconOpen weightsMixture of expertsMemoryhttps://llmobile.news/ticker/iphone-18-pro-a20-pro-neural-engine/https://llmobile.news/ticker/iphone-18-pro-a20-pro-neural-engine/Wed, 09 Sep 2026 19:00:00 +0200Apple has introduced the iPhone 18 Pro and iPhone 18 Pro Max with the A20 Pro chip. According to Apple, the chip has a new dual 16-core Neural Engine, 32 cores in total, which the company describes as double the AI processing power of A19 Pro. Apple says the Neural Engine accelerates on-device AI models and computational photography. Image: Apple. The A20 Pro also offers 50 percent more memory bandwidth than A19 Pro, and its 6-core CPU includes integrated Neural Accelerators. Apple positions the chip for “more advanced on-device AI workloads” alongside games. The phones ship with iOS 27, Apple Intelligence and the new Siri AI. Apple states that Apple Intelligence uses on-device processing together with Private Cloud Compute. Pre-orders start on 12 September, with availability from 18 September; prices start at $1,199 for the iPhone 18 Pro and $1,299 for the iPhone 18 Pro Max, both with 256 GB. Source: https://www.apple.com/newsroom/2026/09/apple-debuts-iphone-18-pro-and-iphone-18-pro-max/ Read the article: https://llmobile.news/ticker/iphone-18-pro-a20-pro-neural-engine/AppleiPhoneChipsNPUhttps://llmobile.news/ticker/artificial-analysis-mobile-benchmark/https://llmobile.news/ticker/artificial-analysis-mobile-benchmark/Mon, 24 Aug 2026 15:00:00 +0200 Chart: Artificial Analysis. Artificial Analysis has published a benchmark of small language models running locally on an iPhone 17 Pro. The study covers 41 quantised builds, 33 of which ran successfully on the device. Models were executed with llama.cpp at 4-bit quantisation or smaller, and had to fit within 8 GB of memory including the KV cache at 8K context. Artificial Analysis states that the inference benchmarking is operated in partnership with Liquid AI. Nanbeige4.2-3B and LFM2.5-2.6B tied for the highest intelligence score at 63. On speed and memory the two differ sharply: LFM2.5-2.6B completed the test prompt in 8.0 seconds using 2.3 GB, against 21.4 seconds and 4.0 GB for Nanbeige4.2-3B. Ornith-1.0-9B scored 62 and Qwen3.5 9B (Reasoning) 61. End-to-end generation time for a 1,024-token prompt plus 256 output tokens ranged from 0.9 seconds for LFM2.5-230M to 26.7 seconds for Falcon-H1R-7B. Peak in-test memory at 4K context ran from 0.4 GB to 6.9 GB. The intelligence index combines tool calling, instruction following, knowledge, scientific reasoning and mathematics. Artificial Analysis notes that context limits, not parameter count alone, held back several 9B models, which frequently hit a 16K ceiling. Chart: Artificial Analysis. Source: https://artificialanalysis.ai/articles/mobile-phone-intelligence-inference Read the article: https://llmobile.news/ticker/artificial-analysis-mobile-benchmark/BenchmarksiPhoneQuantisationllama.cppLiquid AIhttps://llmobile.news/ticker/mobilemoe/https://llmobile.news/ticker/mobilemoe/Tue, 26 May 2026 19:58:00 +0200Meta AI published MobileMoE on May 26, 2026, a family of three language models for phones that hold 1.3B, 2.8B and 5.3B parameters but run only 272M, 528M and 922M of them on any given token. Each one is a sparse mixture of experts, meaning the feedforward block in every layer is split into 60 small networks plus one always-on shared network, and a router picks 4 of the 60 for each token while the rest sit idle. Per-token compute therefore follows the active count, and the memory footprint follows the total. That split is what makes the models awkward to host on a phone. The authors quantise all three to 4 bits with quantisation-aware training, which puts the weights at 0.68 GB, 1.48 GB and 2.75 GB, because every expert has to be reachable even though only a few run per token. On a Samsung Galaxy S25 they measure peak resident memory during inference at 1.49 GB for the smallest model at an 8k context against 1.91 GB for the dense MobileLLM-Pro, and at 4.71 GB for the largest. Sizing the whole expert set to fit phone DRAM is the opposite of the route Edge0 takes, which streams expert weights off storage to keep a 35B model inside a couple of gigabytes. The paper reports that peak memory under real prompts runs 1.2 to 2.1 times higher than under repeated dummy tokens for the mixture-of-experts models, while staying flat for the dense baseline, because varied input routes to more experts and pulls more of them into RAM. Meta AI therefore profiled with real code, knowledge and maths prompts and calls dummy-prompt figures a lower bound rather than a measurement. The authors ran the models through ExecuTorch on the Galaxy S25 with a Snapdragon 8 Elite on 4 CPU threads and on an iPhone 16 Pro with an A18 Pro on 2 CPU threads, after writing a fused mixture-of-experts operator because mobile CPU backends ship none. On a 512-token prompt they measure the smallest model decoding at 130.0 tok/s on the Galaxy S25 against 56.5 for MobileLLM-Pro, and at 180.5 tok/s on the iPhone 16 Pro against 55.1. Time to first token on a 1k prompt drops from 4.26 s to 2.01 s on the Galaxy and from 6.03 s to 2.14 s on the iPhone. Across both phones, CPU and GPU, and the XNNPACK and MLX backends, Meta AI reports the smallest model running 1.8 to 3.8 times faster at prefill and 2.2 to 3.4 times faster at decode than the dense baseline at comparable 4-bit weight memory. Across 14 benchmarks the authors score the instruction-tuned models at 46.7, 55.3 and 60.1 on average at 16-bit precision, against 42.7 for the dense Qwen3.5 0.8B, 44.0 for Gemma 3 1B, 46.9 for Llama 3.2 1B and 50.8 for Qwen3.5 2B, the last of which carries 1.9B active parameters. The largest model reaches 60.1 against 55.6 for Ai2’s OLMoE-1B-7B, with 30 percent fewer active and 23 percent fewer total parameters. After 4-bit quantisation those averages fall to 44.0, 52.5 and 57.8, which leaves the smallest model 1.5 points behind MobileLLM-Pro at 45.5 while scoring 9.4 points higher on MMLU. Meta AI states that the training data across all four stages comes from openly licensed public datasets, names no weight release, and lists mobile NPU deployment, expert pruning and mixed-precision quantisation as further work. Chart: Meta AI. The authors' own comparison of instruction-tuned models at 16-bit precision, with the horizontal axis giving total parameters alongside the 4-bit weight footprint those imply. Source: https://arxiv.org/abs/2605.27358 Read the article: https://llmobile.news/ticker/mobilemoe/MetaMixture of expertsMemoryQuantisationiPhonehttps://llmobile.news/ticker/llm-inference-at-the-edge/https://llmobile.news/ticker/llm-inference-at-the-edge/Tue, 24 Mar 2026 19:28:00 +0100Four authors at Conscious Engines published a benchmark of Qwen 2.5 1.5B at 4-bit across four edge platforms on March 24, 2026, feeding each device the same 258-token prompt 20 times in a row with a 1-second gap instead of once. On an iPhone 16 Pro the first two runs peak at 40.49 tokens per second, and from run 17 onwards the phone holds a plateau of 23.67 tok/s, which the authors put at 41.5 percent below that peak. The iPhone passes through the three thermal states iOS reports to an app. Runs 1 and 2 sit in Normal and average 36.83 tok/s, runs 3 to 16 sit in Warm and average 24.96 tok/s, and runs 17 to 20 sit in Hot at 23.67 tok/s with a variation of 1.2 percent. The authors write that the 1-second gap allows no thermal recovery, and they measure 5 percent of battery drained over the 20 runs, which they project to roughly 400 replies of this length per charge. A Samsung Galaxy S24 Ultra degrades more gently in the same test. Its Adreno 750 GPU boosts to 1,000 MHz on runs 4 and 5 for a peak of 12.21 tok/s, then steps down to a 720 to 770 MHz plateau from run 8 where the authors measure 10.38 tok/s, 15 percent off the peak, with a low of 9.55 tok/s on the final run. They stress that reaching this plateau took deliberate tuning, with the display off, the context capped at 2,048 tokens and MLC-LLM’s prefill chunk cut to 128 tokens, and that earlier unconstrained runs hit an OS-enforced GPU frequency floor. Neither phone ran the model on its NPU. The paper states that MLX dispatches to the A18 Pro GPU through Metal and does not target Apple’s Neural Engine, and that the S24 Ultra ran through MLC-LLM on the Adreno GPU, leaving the Snapdragon 8 Gen 3’s Hexagon NPU outside the measurement. The one NPU that did run the model, a Hailo-10H module in a Raspberry Pi 5, was also the only part that never throttled, holding 6.914 tok/s across all 20 runs with a variation of 0.04 percent at under 2 W, its CPU at 52.7°C and the NPU at 58.5°C, with attention layers that hailo-ollama does not offload falling back to the Pi’s Cortex-A76 CPU. A user running llama.cpp’s Hexagon NPU backend on a Snapdragon 8 Gen 3 phone reported 12.5 tok/s of generation on Gemma 3 4B and said the phone stayed cool. A laptop RTX 4050 on battery led on speed at 131.7 tok/s and 34.12 W, its GPU climbing from 55°C to 70°C without throttling. Energy per token comes out at 270.5 mJ for the Hailo module measured across the whole system, 297.3 mJ for the RTX 4050 measured at the GPU, and 143.0 mJ for the S24 Ultra plateau derived from the Android fuel gauge with the display off. The authors state that the three readings differ in scope and call the resemblance suggestive rather than settled, and report no energy figure for the iPhone because iOS exposes no per-component power reading. They tested one model, one prompt and one unit per platform across four different runtimes, and describe the results as characterising hardware and software together rather than hardware on its own. Source: https://arxiv.org/abs/2603.23640 Read the article: https://llmobile.news/ticker/llm-inference-at-the-edge/BenchmarksNPUiPhoneSamsungResearchhttps://llmobile.news/ticker/a19-pro-neural-accelerators/https://llmobile.news/ticker/a19-pro-neural-accelerators/Tue, 09 Sep 2025 20:00:00 +0200Apple announced the A19 Pro alongside the iPhone 17 Pro on September 9, 2025. Its 6-core GPU carries Neural Accelerators built into each core, which Apple says work together with the 16-core Neural Engine to power AI models, graphics and games. The company states that the chip enables running large local language models on the phone. The A19 Pro also has a 6-core CPU that Apple calls the fastest in any smartphone, along with a larger cache and more memory than the A18 Pro. Combined with a vapor chamber cooling system, Apple puts sustained performance at up to 40 percent above the previous generation. Image: Apple. The vapor chamber Apple credits for the sustained performance gain. Source: https://www.apple.com/newsroom/2025/09/apple-unveils-iphone-17-pro-and-iphone-17-pro-max/ Read the article: https://llmobile.news/ticker/a19-pro-neural-accelerators/AppleiPhoneChipsApple SiliconNPUhttps://llmobile.news/ticker/locally-ai/https://llmobile.news/ticker/locally-ai/Sun, 13 Apr 2025 09:00:00 +0200Adrien Grondin released Locally AI on April 13, 2025, an iPhone and iPad app that downloads open-weight language models and runs them entirely on the device. The App Store listing gives that release date and lists the app as free with no in-app purchases. Once a model is on disk the app needs no account and no internet connection, and Grondin states that no data is collected and nothing is sent to a cloud service. The site at launch named Meta Llama 3.2 and Llama 3.1, Google Gemma 2, Qwen 2 and DeepSeek R1 as the models on offer, and described them as language and vision models, so a picture can go into the prompt alongside text. A user picks a model, downloads it once and can edit the system prompt the model runs under. Grondin builds the app on MLX, Apple’s machine learning framework for Apple silicon, and attributes the performance to unified memory, where the CPU and the GPU reach the same memory pool so arrays need not be copied between them. The app fetches open-weight model files itself rather than calling any model that ships with the operating system. Neither the site nor the App Store entry names a minimum chip. The site says only that Locally AI covers recent iPhone and iPad models and is optimised for Apple silicon, and the listing asks for iOS 18.1 or iPadOS 18.1 or later. No generation speed, memory requirement or model file size is given anywhere in the developer’s own material, and the launch site was a bought template whose team, testimonial and FAQ sections still carried placeholder Latin text. Update, April 8, 2026. Grondin added support for Apple’s on-device Foundation Models framework in September 2025, so the app can open a conversation with the model built into iOS 26 without downloading anything, which Simon Willison recorded on September 21, 2025. A Mac version followed and requires macOS 26 or later. LM Studio announced on April 8, 2026 that Locally AI had joined the company and that Grondin had moved with it, and the App Store entry now reads Locally AI by LM Studio with Element Labs Inc as the seller. Source: https://locallyai.app/ Read the article: https://llmobile.news/ticker/locally-ai/AppleApple SiliconMLXiOSiPhonehttps://llmobile.news/ticker/huggingsnap/https://llmobile.news/ticker/huggingsnap/Wed, 19 Mar 2025 16:16:00 +0100Hugging Face published HuggingSnap, an iPhone app that describes what the camera is pointed at, with the model running on the phone rather than in the cloud. The app’s runtime configuration loads SmolVLM2 at 500M parameters in Apple’s MLX format. Hugging Face had trailed the app in its SmolVLM2 announcement on February 20, 2025, writing that the 500M model lets users analyse and understand video content directly on their device with no cloud required. The README describes the app as a way to point the camera and have text translated or summarised, identify plants and animals and learn more about surrounding objects. Inference runs through MLX Swift, Apple’s machine learning framework for Apple silicon, with a modified fork of Apple’s MLX example code supplying the vision-language support. The app pulls the weights from the Hugging Face Hub the first time it runs, then holds Metal memory to 82 percent of what iOS reports as available to the process. Input comes from the live camera, from stills and clips captured in the app, and from photos and videos picked out of the library. A tap on the shutter takes a picture and a press and hold records video, after which the app offers a describe action for images and a summarise action for video, with questions typed or dictated. The app’s own interface states that photos and videos are processed entirely on the device and that no data is sent to the cloud. The README states that installation needs an iPhone running iOS 18, and the Xcode project sets iOS 18.0 as the deployment target and iPhone as the only device family, with no Mac Catalyst or visionOS build. TechCrunch reported the app as a free App Store download on March 19, 2025, quoting the listing as saying it works offline and processes all data on the phone. The SmolVLM2 weights are Apache 2.0, while the app repository itself carries no licence file. Source: https://github.com/huggingface/HuggingSnap Read the article: https://llmobile.news/ticker/huggingsnap/Hugging FaceiOSiPhoneApple SiliconOpen weightshttps://llmobile.news/ticker/exolabs-benchmarks/https://llmobile.news/ticker/exolabs-benchmarks/Wed, 25 Dec 2024 19:55:00 +0100EXO Labs opened a public benchmark site on December 25, 2024, as the first entry in a twelve-part series, publishing generation speed and time to first token measured on real consumer devices. EXO states these are measurements rather than estimates. In the data behind the site, an iPhone 15 Pro with 8 GB of RAM generates 14.0 tok/s on Llama 3.1 8B and takes 1.67 seconds to return its first token. A Galaxy S24 Ultra with 12 GB manages 5.6 tok/s on the same 8B model, with 4.16 seconds to the first token, according to EXO’s published results. On the smaller Llama 3.2 1B the iPhone reaches 56.1 tok/s and the Galaxy 22.5 tok/s. A single Mac mini M4 Pro with 24 GB sits at 49.3 tok/s on Llama 3.1 8B, which puts the phone at under a third of the Mac’s rate on the same model. EXO splits a model into contiguous slices of its layers and hands each slice to a different device, so a model too large for one machine can run across several. The activations passed from one device to the next are small, under 4 KB for Llama 3.2 3B, and EXO says the bottleneck is network latency rather than bandwidth. Pooling phones does not speed up a single reply. A cluster of two iPhone 15 Pros and one Galaxy S24 Ultra runs Llama 3.1 8B at 7.0 tok/s, below the 14.0 tok/s of one iPhone alone, which EXO attributes to network overhead when the model already fits on a single device. Three Mac mini M4 Pros run Llama 3.3 70B at 4.1 tok/s, a model that does not fit in the 24 GB of any one of them. Every entry carries a timestamp, a git commit and the identifier of the automated run that produced it, and the site is free to browse. EXO says it is expanding the suite towards mixed device types and different quantisations, and the exo runtime itself is on GitHub under Apache 2.0. Source: https://blog.exolabs.net/day-1/ Read the article: https://llmobile.news/ticker/exolabs-benchmarks/BenchmarksiPhoneAndroidLlamaApple Siliconhttps://llmobile.news/ticker/palmbench/https://llmobile.news/ticker/palmbench/Sat, 05 Oct 2024 05:37:00 +0200Researchers at the University of Wisconsin-Madison, Amazon Web Services AI and Uber ran quantised language models across eight phones and edge boards and measured the widest gap between Apple and everything else. In PalmBench, published as a conference paper at ICLR 2025, TinyLlama-1.1B at 4-bit reached close to 100 tok/s on an iPhone 15 Pro against roughly 30 tok/s on the fastest Android phone in the set, a Pixel 7. The authors report that the three-year-old iPhone 12 Pro also stayed ahead of every Android device and of Nvidia’s Jetson Orin Nano. Chart: Li et al., licensed CC BY 4.0. Throughput measured with MLC LLM, which was the only framework that ran on all eight platforms. The test bed covers a Pixel 4, Pixel 5a, Pixel 7, Galaxy S22 Ultra, iPhone 12 Pro, iPhone 15 Pro, an Orange Pi 5 and a Jetson Orin Nano, each with 6 GB or 8 GB of memory. Models include Llama-2-7B, Llama-3 and 3.1-8B, Llama-3.2-3B, Mistral-7B, Vicuna, Qwen2, Phi-2, Gemma-2-2B, RedPajama-INCITE-3B and TinyLlama-1.1B, quantised from 2-bit to 8-bit and additionally with GPTQ, AWQ and FasterTransformer. The team used two runtimes, MLC LLM and llama.cpp, and states that llama.cpp does not run on iPhone, so the cross-platform numbers come from MLC. Quantisation costs more answer quality than the bit count alone suggests. Scoring each quantised build’s exact match against the unquantised model’s own answers on SQuAD and Natural Questions, the paper’s figure puts 4-bit builds between about 50 and 61 percent, 6-bit between roughly 62 and 78 percent, and 2-bit and 3-bit builds as low as 15 percent. The authors note that 3-bit did not clearly beat 2-bit, and that 5-bit landed close to 4-bit while consuming more resources and degrading more than any 4-bit variant. Chart: Li et al., licensed CC BY 4.0. Exact match is measured against the answers of the same model before quantisation, not against ground truth. The 4-bit builds drew more power than the 3-bit ones rather than less. The paper’s table records 4-bit consuming more energy on all eight devices for the same 3B Llama model, rising from 10.13 to 13.05 mWh on the iPhone 15 Pro and from 14.54 to 14.68 mWh on the Pixel 7, which the authors put down to the higher-precision build occupying more GPU duty cycles. Surface temperature measured with a FLIR C5 thermal camera peaked at 75.4 °C on the Orange Pi 5 and 46.3 °C on the iPhone 15 Pro at 4-bit. Harmful output tracked the bit count more cleanly, with the hallucination rate on Llama-3-8B at 34.7 percent for 2-bit against 9.1 percent for 4-bit GPTQ and 7.9 percent at 8-bit. The authors name several limits on their own work. They cannot compare the two runtimes on iOS because llama.cpp does not support the iPhone, they profiled total memory only at 3-bit and 4-bit because higher-bit MLC builds exceeded the devices’ capacity, and they dropped models that would not fit at all, such as the unquantised Llama-2-7B at 13.11 GB. They also note that both runtimes expose few decoding options, with beam search unavailable in llama.cpp. Artificial Analysis has since run an independent benchmark of 33 quantised models on a single iPhone 17 Pro. Source: https://arxiv.org/abs/2410.05315 Read the article: https://llmobile.news/ticker/palmbench/BenchmarksQuantisationiPhoneAndroidMLC LLMhttps://llmobile.news/ticker/mobileclip/https://llmobile.news/ticker/mobileclip/Fri, 21 Jun 2024 02:15:00 +0200Five Apple researchers presented MobileCLIP at CVPR 2024 on June 20, a family of image-text models built for phones. They exported each model with Core ML Tools 7.0 and timed it at batch size 1 on an iPhone 12 Pro Max running iOS 17.0.3, and their Table 7 gives the smallest variant, MobileCLIP-S0, 1.5 ms for the image encoder and 1.6 ms for the text encoder at 67.8% zero-shot accuracy on ImageNet. The same authors measure OpenAI’s ViT-B/16 CLIP on that phone at 11.5 ms and 3.3 ms for 68.3%, and both models average 58.1 across the 38 datasets of the DataComp evaluation suite. Multi-modal reinforced training, the method the paper introduces, moves the expensive part of distillation out of the training loop and into the dataset. Apple reinforces the DataComp dataset once, writing five synthetic captions per image with the CoCa model in OpenCLIP, storing the parameters of strong random image augmentations, 30 per image for the 12M subset and 10 for the billion-sample set, and storing the embeddings that two ViT-L/14 teacher models produce for those augmented images and for the real and synthetic captions. Training afterwards reads the captions and the teacher embeddings from disk, so the captioning model and the teachers never run again however many models are trained on the set. The paper puts numbers on what that saves. One epoch over the 12.8M samples of the reinforced set on a single node of 8 A100-80GB GPUs takes 1.3 hours, the same as an epoch on the unreinforced data, against 21.1 hours when the captioning model and the teacher ensemble run during training, according to Table 4d. The cost moves to storage, and Table 4c puts DataComp-1B at 90 TB against 140 TB for the reinforced DataCompDR-1B, with the 12M subset going from 0.9 TB to 1.9 TB. MobileCLIP comes in four sizes, and Table 7 lists image and text encoder parameters separately. S0 pairs an 11.4M-parameter image encoder with a 42.4M-parameter text encoder, S1 uses 21.5M and 63.4M at 2.5 ms and 3.3 ms, S2 uses 35.7M and 63.4M at 3.6 ms and 3.3 ms, and MobileCLIP-B keeps a standard ViT-B/16 image encoder at 86.3M and 10.4 ms. Zero-shot ImageNet accuracy across the four runs 67.8%, 72.6%, 74.4% and 76.8%, and a version of B trained on 39B seen samples instead of 13B reaches 77.2%. All three small variants replace the vision transformer with a convolution-transformer hybrid derived from Apple’s FastViT, and S0 also swaps the text encoder for one that mixes 1-D convolutions with four self-attention layers, which the authors report as 5% smaller and 15.8% faster than the 12-layer transformer text encoder that S1 and S2 keep. Apple released the code, the weights and the reinforced data at apple/ml-mobileclip, with checkpoints for all four variants and the longer-trained B, and with DataCompDR-1B and DataCompDR-12M on Hugging Face as synthetic captions, augmentation parameters and teacher embeddings keyed to DataComp images rather than as the images themselves. The repository lists the code under MIT, the models under Apple’s own machine learning research model terms, and the data under CC BY-NC-ND. Apple showed zero-shot scene classification running in real time on an iPhone in the CVPR exhibit hall, and added an iOS app for the same task to the repository in November 2024. Source: https://openaccess.thecvf.com/content/CVPR2024/html/Vasu_MobileCLIP_Fast_Image-Text_Models_through_Multi-Modal_Reinforced_Training_CVPR_2024_paper.html Read the article: https://llmobile.news/ticker/mobileclip/AppleiPhoneResearchBenchmarksOpen weightshttps://llmobile.news/ticker/apple-intelligence-foundation-models/https://llmobile.news/ticker/apple-intelligence-foundation-models/Mon, 10 Jun 2024 21:00:00 +0200Apple introduced Apple Intelligence on June 10, 2024, built on a roughly 3-billion-parameter language model that runs on the device and a larger model that runs on Apple silicon servers through Private Cloud Compute. The on-device model uses a 49K token vocabulary, the server model 100K. Apple compresses the on-device model with a mix of 2-bit and 4-bit weights, averaging 3.7 bits per weight, and says an option at 3.5 bits per weight costs no significant quality. On an iPhone 15 Pro the company reports a time to first token of about 0.6 milliseconds per prompt token and 30 tokens per second, both measured before token speculation. Rather than ship a separate model per feature, Apple swaps LoRA adapters on top of the shared base model. The adapters are stored in 16 bits and, at rank 16, take tens of megabytes each. Apple says human graders preferred its models over comparable ones including GPT-3.5, GPT-4, Llama, Mistral and Gemma across its evaluations. Diagram: Apple. Source: https://machinelearning.apple.com/research/introducing-apple-foundation-models Read the article: https://llmobile.news/ticker/apple-intelligence-foundation-models/AppleiPhoneiOSQuantisationApple Siliconhttps://llmobile.news/ticker/phi-3-mini/https://llmobile.news/ticker/phi-3-mini/Mon, 22 Apr 2024 18:00:00 +0200Microsoft introduced Phi-3-mini on April 22, 2024, a 3.8-billion-parameter model trained on 3.3 trillion tokens. The technical report puts it at 69 percent on MMLU and 8.38 on MT-bench, which the authors say rivals models such as Mixtral 8x7B and GPT-3.5. Quantised to 4 bits the model occupies about 1.8 GB, small enough for a phone. The report describes the test directly. We tested the quantized model by deploying phi-3-mini on iPhone 14 with A16 Bionic chip running natively on-device and fully offline achieving more than 12 tokens per second. Screenshots: Microsoft, Figure 1 of the technical report. The same report covers two larger siblings, Phi-3-small at 7B parameters and Phi-3-medium at 14B, both trained on 4.8 trillion tokens. Microsoft reports 75 percent on MMLU for the small model and 78 percent for the medium one. Source: https://arxiv.org/abs/2404.14219 Read the article: https://llmobile.news/ticker/phi-3-mini/MicrosoftPhiiPhoneQuantisationOpen weightshttps://llmobile.news/ticker/mobile-llms/https://llmobile.news/ticker/mobile-llms/Thu, 22 Feb 2024 19:58:00 +0100Meta Reality Labs researchers published MobileLLM on February 22, 2024, a family of language models built to run on a phone rather than in a data centre. The paper argues that below a billion parameters the shape of the network matters more than the parameter count or the amount of training data. Its 125M and 350M models beat the previous best models at those sizes by 2.7 and 4.3 points on eight zero-shot common sense reasoning tasks. The central finding is that depth beats width. The authors trained 19 models of roughly equal size but different proportions, 9 around 125M parameters and 10 around 350M, and report that 30-layer and 42-layer designs scored higher than the 12-layer shape used by OPT, GPT-Neo and Pythia at the same size. Sharing one weight matrix between the input embedding and the output layer removes 16M parameters, about 11.8% of a 125M model, for a 0.2 point accuracy drop the authors then recover by spending the freed budget on two more layers. Grouped query attention, where several query heads read one shared set of key and value heads, cuts 16 key-value heads to 4 at close to the same accuracy. The paper sets out what a phone can actually spare. The authors put DRAM on current handsets at 6 to 12 GB, shared with the operating system and every other app, and argue that an app should not claim more than 10% of it. For energy they use a rule of thumb of 0.1 J per token per billion parameters, which puts a 7B model at 0.7 J/token, drains 0.2% of an iPhone battery every 64 tokens and allows under 2 hours of conversation at 10 tok/s. A 350M model with 8-bit weights costs 0.035 J/token by the same arithmetic, which the authors say covers a full day of use. Meta also profiled the models on hardware rather than estimating. Running the 125M model in FP16 through ExecuTorch on an iPhone 13 with iOS 17.2.1 and the Metal Performance Shaders backend, the authors measured 39.2 ms to load, 1361.7 ms to initialise and 15.6 ms to execute, the last averaged over 50 runs. MobileLLM-LS, the layer-sharing variant, computes each transformer block twice in a row so that 30 blocks of weights behave like 60 layers at the same file size, a pattern the team picked over the alternatives because the repeated weights stay in the roughly 20 MB of on-chip cache instead of being fetched from DRAM again. That costs 2.2% more loading and initialisation time and 2.6% more execution time, against 143% and 86% for a model that really has 60 layers, and adds 0.7 and 0.8 points of accuracy at 125M and 350M. After fine-tuning, MobileLLM-350M scores 3.28 on the MT-Bench chat benchmark against 1.37 for OPT-350M and 2.24 for the 1.3B OPT, and the layer-shared 350M wins 48.2% of AlpacaEval comparisons against text-davinci-001, a model that wins 50% against itself. For API calling the team built a synthetic set of 5,000 training and 2,500 test conversations that turn requests such as setting an alarm into a function call, and MobileLLM-350M scored 65.3 on intent exact match against 62.8 for Llama 2 7B, while trailing the larger model on the structure of the call and on the wording of the reply. Quantising weights and activations to 8 bits costs under 0.5 points. The work was published at ICML 2024 and the code is in Meta’s repository. Source: https://arxiv.org/abs/2402.14905 Read the article: https://llmobile.news/ticker/mobile-llms/MetaResearchMemoryiPhoneBenchmarkshttps://llmobile.news/ticker/llmfarm/https://llmobile.news/ticker/llmfarm/Wed, 14 Jun 2023 18:14:00 +0200Artem Savkin published the first release of LLMFarm on June 14, 2023, an iOS and macOS app that loads a quantised model file kept on the device and holds a chat with it offline. The app is built on ggml and llama.cpp, and the README asks for iOS 16 or later and macOS 13 or later. Screenshot from the first release. Screenshot: LLMFarm. The app reads two llama.cpp file formats, the older ggjtv3 files with a .bin extension and GGUF files, according to the project FAQ. The first release covered LLaMA, GPT-NeoX and GPT-2 style models, with 3B builds of OpenLLaMa, StableLM and Dolly v2 among those it listed as working on an iPhone. The README has since added Gemma, Phi, Qwen, Mixtral, RWKV and LLaVA-style vision models, and marks most of them as running through Metal, Apple’s GPU interface, which it says does not work on Intel Macs. Quantisation happens before a model reaches the phone, and the project’s on-device notes list 7B Llama 2 and Orca builds at 3-bit and 4-bit as the ones tried on an iPhone 12 Pro Max, with no published token rates. Each chat writes a JSON file of inference options, which the first release documented with defaults of temperature 0.8, top-k 40, top-p 0.95, a 2048-token context and a repeat penalty of 1.1. The app also offers tail free sampling, locally typical sampling, Mirostat, greedy decoding and grammar-constrained output. Models arrive either through an add model button that imports a file from local storage or through a download list inside the app, and the sandboxed app keeps them in its own models directory. Memory sets the ceiling on an iPhone. The first release’s model table marked 3B models as working on an iPhone 12 Pro Max and 7B and 13B models as returning a “bad alloc” error, and the README said extended memory might be needed to solve it. The FAQ states that the Extended Virtual Addressing and increased-memory-limit entitlements are required to run models larger than 3B, and that the most common reason for a crash is lack of memory. LLMFarm is published under the MIT license, and the README now says the app is temporarily unavailable in TestFlight and the App Store. Source: https://github.com/guinmoon/LLMFarm Read the article: https://llmobile.news/ticker/llmfarm/llama.cppiOSiPhoneOpen sourceQuantisationhttps://llmobile.news/ticker/mlc-llm-consumer-devices/https://llmobile.news/ticker/mlc-llm-consumer-devices/Mon, 01 May 2023 18:00:00 +0200The MLC project released MLC LLM on May 1, 2023, a compiler and runtime that deploys language models natively on phones, browsers and consumer hardware rather than serving them from a data centre. The team pointed users to a TestFlight build to try the chat app on iOS. Your browser does not support this video. This video could not be loaded. Use the link below to open it directly. Video: MLC. The iOS app generating a reply on the phone. Open the MLC post MLC LLM compiles models through the Apache TVM Unity stack and maps them to the Vulkan and Metal graphics APIs, which the team says covers the majority of consumer platforms. Targets named at launch include Windows, Linux, macOS and iOS, plus AMD, NVIDIA, Intel and Apple GPUs and a Steam Deck. A companion project, WebLLM, offloads the same models into web browsers through WebGPU. Three weeks later the team added RedPajama-3B across the same set of targets, with a browser demo and a command line tool for desktop machines. The code is on GitHub under the Apache 2.0 license. The project now exposes an OpenAI-compatible API through a REST server as well as Python, JavaScript, iOS and Android bindings, and covers Android GPUs via OpenCL alongside CUDA and ROCm on the desktop. Source: https://blog.mlc.ai/2023/05/01/bringing-accelerated-llm-to-consumer-hardware Read the article: https://llmobile.news/ticker/mlc-llm-consumer-devices/MLC LLMiOSiPhoneOpen sourceWebGPUhttps://llmobile.news/ticker/distilbert/https://llmobile.news/ticker/distilbert/Wed, 02 Oct 2019 19:56:00 +0200Hugging Face published DistilBERT on October 2, 2019, a compressed version of BERT trained by knowledge distillation, where a small student model learns to reproduce the output of a larger teacher. The student carries 66M parameters against BERT-base’s 110M, and Victor Sanh and his co-authors report it retaining 97% of BERT’s score on the GLUE language understanding benchmark while running 60% faster. The authors tested the model on a phone by building a question-answering app for an iPhone 7 Plus and comparing it against their own BERT-base question-answering model. Excluding the tokenisation step, they measured DistilBERT at 71% faster, and the whole model in that app weighs 207 MB, which they note could shrink further with quantisation. On a server CPU at batch size 1, the same pair takes 410 seconds against 668 for a full pass over the STS-B development set. Three losses train the student at once. A distillation loss pushes it towards the teacher’s full output distribution rather than only the correct token, a masked language modelling loss keeps BERT’s own pre-training objective in place, and a cosine embedding loss aligns the directions of student and teacher hidden state vectors. The student halves BERT’s layer count, drops the token-type embeddings and the pooler, and starts from every second layer of the teacher, an initialisation the ablation study values at 3.69 points of GLUE macro score. Training ran on eight 16 GB V100 GPUs for about 90 hours on English Wikipedia and the Toronto Book Corpus. Hugging Face released the trained weights together with the training code in its transformers library, and published the iOS demo separately as Core ML code. The paper was accepted at the fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing, co-located with NeurIPS 2019. Source: https://arxiv.org/abs/1910.01108 Read the article: https://llmobile.news/ticker/distilbert/ResearchDistillationiPhoneOpen weights