Research
73 updates on Research.
PowerInfer-2 runs a 47B model on a OnePlus 12 at 11.68 tokens per second
Shanghai Jiao Tong University researchers report a 47B model decoding at 11.68 tokens per second on a OnePlus 12, with weights streamed from flash.
BUPT proposes one 9.2B model in the OS that all apps call through adapters
A BUPT team proposes that the phone OS ship one 9.2B multimodal model all apps call through small adapters, and matched app models on 85% of 50 datasets.
Apple releases OpenELM at 270M to 3B, with parameters spread unevenly across layers
Apple published four models from 270M to 3B parameters with the full training framework and code to run them through MLX on Apple silicon.
Octopus v3 picks an action from an image and a query in under 1B parameters
A technical report describes a sub-billion multimodal agent that turns an image and a text query into a function call, shown through ten worked examples.
Octopus v2 is a 2B model that calls Android APIs with one token per function
A 2B Gemma fine-tune gives every Android API its own token, and the authors report 99.524% accuracy and 0.38 seconds per call, ahead of GPT-4.
Octopus fine-tunes a 2B model to 93 percent on API function calls
Stanford and Harvard authors fine-tuned open 2B to 7B models on 20,000 RapidAPI functions and report up to 97 percent function call accuracy.
One shared on-device LLM keeps a context per app and switches in 0.27 seconds
Peking University, BUPT and Tsinghua researchers compress and swap each app conversation context in 16-token chunks, switching up to 20 times faster.
MeRino designs sub-100M language models that run 4.9 times faster than OPT-350M
Researchers design 52M to 64M parameter transformers by maximising entropy under a compute budget, matching OPT-350M accuracy on an NVIDIA Jetson Nano.
Stability AI trains Stable LM 2 1.6B on seven languages and 2 trillion tokens
The technical report details a 1.6B model pre-trained on seven languages and measures 127 tok/s for a 4-bit build on an M2 Mac mini.
BitNet b1.58 gives every weight three values and runs 3B in 2.22 GB
Microsoft trained models whose every weight is -1, 0 or 1, which replaces multiplication with addition, and reports parity with full-precision Llama from 3B.
MobiLlama is a fully transparent 0.5B model that runs in 770 MB on a phone
MBZUAI published a 0.5B model that shares one feed-forward block across all layers and reports 7.02 tok/s in 770 MB on a Snapdragon 685 phone.
Meta's MobileLLM trades width for depth and gains 2.7 and 4.3 points
Meta Reality Labs built 125M and 350M models around deep and thin layers and profiled them on an iPhone 13 through ExecuTorch.