LLMobile.news

Quantisation

46 updates on Quantisation.

  1. One shared on-device LLM keeps a context per app and switches in 0.27 seconds

    Peking University, BUPT and Tsinghua researchers compress and swap each app conversation context in 16-token chunks, switching up to 20 times faster.

  2. BitNet b1.58 gives every weight three values and runs 3B in 2.22 GB

    Microsoft trained models whose every weight is -1, 0 or 1, which replaces multiplication with addition, and reports parity with full-precision Llama from 3B.

  3. MobiLlama is a fully transparent 0.5B model that runs in 770 MB on a phone

    MBZUAI published a 0.5B model that shares one feed-forward block across all layers and reports 7.02 tok/s in 770 MB on a Snapdragon 685 phone.

  4. Arm runs Llama 2 7B at 9.6 tokens per second on three mobile CPU cores

    Arm demonstrated Llama 2 7B generating 9.6 tokens per second on three Cortex-A700 CPU cores, using int4 kernels it says beat llama.cpp by 20 percent.

  5. MIT HAN Lab publishes TinyChatEngine for 4-bit LLMs on laptops and Raspberry Pi

    The from-scratch C and C++ engine from Song Han's group at MIT runs 4-bit LLaMA2-7B on x86, Apple silicon, Raspberry Pi and Nvidia GPUs.

  6. LLMFarm runs llama.cpp models offline on iOS and macOS

    The iOS and macOS app loads a quantised model file off the device and chats with it offline, with 3B the practical size on an iPhone.

  7. Qualcomm argues for hybrid AI, citing 10 times the cost per generative AI search query

    Qualcomm argues that cloud-only inference cannot scale, and puts models of 1B to 10B parameters on phones and laptops at INT4.

  8. Georgi Gerganov publishes llama.cpp, LLM inference in plain C and C++

    The dependency-free C and C++ inference engine supports 1.5-bit to 8-bit quantisation and treats Apple silicon as a first-class target.

  9. Qualcomm runs Stable Diffusion on an Android phone for the first time

    Qualcomm AI Research generated 512x512 images in under 15 seconds on a Snapdragon 8 Gen 2 phone after quantising the model from FP32 to INT8.

  10. ONNX Runtime 1.5 adds a mobile build that ships only the operators a model uses

    The release introduced a reduced-size runtime for phones and the ORT model format, the base for the NNAPI, Core ML, QNN and XNNPACK mobile packages.