LLMobile.news

Developer tools

22 updates on Developer tools.

  1. Google ships an experimental MediaPipe LLM Inference API for web, Android and iOS

    The experimental API runs Gemma 2B, Phi 2, Falcon 1B and Stable LM 3B fully on device, with int8 and int4 weights.

  2. Qualcomm AI Hub opens 75 pre-optimised models and profiling on hosted Snapdragon phones

    Qualcomm opened a library of more than 75 models tuned for Snapdragon, with compilation and profiling on real phones in its cloud and two 7B chat models listed.

  3. Gemini Nano ships on the Pixel 8 Pro and Android gets AICore

    Google put Gemini Nano on the Pixel 8 Pro for Recorder summaries and Gboard Smart Reply, and introduced AICore as the Android service behind it.

  4. Apple publishes MLX, where CPU and GPU share arrays without copies

    Apple machine learning research released an array framework for Apple silicon with a unified memory model, lazy evaluation and Swift bindings for iOS.

  5. LLM.swift wraps llama.cpp for on-device text generation in Swift apps

    A Swift package that wraps llama.cpp so iOS, macOS, watchOS, tvOS and visionOS apps can load a GGUF file and generate text on the device.

  6. MIT HAN Lab publishes TinyChatEngine for 4-bit LLMs on laptops and Raspberry Pi

    The from-scratch C and C++ engine from Song Han's group at MIT runs 4-bit LLaMA2-7B on x86, Apple silicon, Raspberry Pi and Nvidia GPUs.

  7. Hugging Face publishes swift-transformers for Core ML models in Swift apps

    A Swift package with tokenizers, Hub downloads and a Core ML generation loop, demonstrated with Llama 2, Falcon, StarCoder and GPT models.

  8. Georgi Gerganov publishes llama.cpp, LLM inference in plain C and C++

    The dependency-free C and C++ inference engine supports 1.5-bit to 8-bit quantisation and treats Apple silicon as a first-class target.

  9. George Hotz starts tinygrad, a framework that ports to an accelerator in about 25 ops

    The tiny corp framework caps its repository at 26,500 lines, ships Metal, Adreno and WebGPU backends, and runs openpilot on a Snapdragon 845 GPU.

  10. ONNX Runtime 1.5 adds a mobile build that ships only the operators a model uses

    The release introduced a reduced-size runtime for phones and the ORT model format, the base for the NNAPI, Core ML, QNN and XNNPACK mobile packages.