Runtimes
5 updates on Runtimes.
Nvidia releases Nemotron-3-Diarization, a 100M speaker model with 0.32 second latency
Nemotron-3-Diarization labels up to eight speakers in streaming or offline audio and ranks first on Voice Arena’s Diarization-Bench v1 with 14.7% DER.
Google releases ADK for Kotlin 1.0 with on-device agents for Android
ADK for Kotlin 1.0 adds Android modules that run agents on the device with Gemma through LiteRT-LM and Gemini Nano through ML Kit.
S2-MoE speeds up MoE decoding by up to 5.3x in llama.cpp
S2-MoE is a self-speculative decoding method for mixture-of-experts models that reaches up to 5.3x speedup and about 2.0x on average.
MobiBench paper benchmarks small LLMs in llama.cpp on two laptops and an RTX 3060
MobiBench is a llama.cpp benchmark suite for speed, memory and accuracy of small LLMs, but the reported tests ran on two laptops and not on phones.
NPUs lead LLM prefill, CPUs lead decoding on Snapdragon phones
A measurement study of five mobile LLM frameworks finds NPUs excel at prefill, CPUs at decoding, and framework gaps of up to 10x on NPUs.