MLC LLM
3 updates on MLC LLM.
lm-Meter times on-device inference and finds prefill, not decode, is the bottleneck
A profiler that runs inside the phone measures each inference phase and reports prefill slowing 158 times as models grow, against 10 times for decode.
PalmBench finds iPhones running local LLMs about three times faster than Android phones
A benchmark of quantised LLMs on eight phones and boards reports throughput, memory, power and heat, with iPhones ahead and 4-bit drawing more power than 3-bit.
MLC LLM brings local language models to iPhone, browsers and consumer GPUs
A compiler stack built on Apache TVM deploys chat models natively to iOS, browsers and consumer GPUs, with an iPhone build handed out through TestFlight.