Developer tools
17 updates on Developer tools.
MIT HAN Lab publishes TinyChatEngine for 4-bit LLMs on laptops and Raspberry Pi
The from-scratch C and C++ engine from Song Han's group at MIT runs 4-bit LLaMA2-7B on x86, Apple silicon, Raspberry Pi and Nvidia GPUs.
Hugging Face publishes swift-transformers for Core ML models in Swift apps
A Swift package with tokenizers, Hub downloads and a Core ML generation loop, demonstrated with Llama 2, Falcon, StarCoder and GPT models.
Georgi Gerganov publishes llama.cpp, LLM inference in plain C and C++
The dependency-free C and C++ inference engine supports 1.5-bit to 8-bit quantisation and treats Apple silicon as a first-class target.
George Hotz starts tinygrad, a framework that ports to an accelerator in about 25 ops
The tiny corp framework caps its repository at 26,500 lines, ships Metal, Adreno and WebGPU backends, and runs openpilot on a Snapdragon 845 GPU.
ONNX Runtime 1.5 adds a mobile build that ships only the operators a model uses
The release introduced a reduced-size runtime for phones and the ORT model format, the base for the NNAPI, Core ML, QNN and XNNPACK mobile packages.