FunctionGemma 270M fine-tune reaches 76.5% on 15 mobile actions, with an Android demo
Google’s FunctionGemma 270M-it reaches 76.5% end-to-end accuracy on mobile device commands once it is fine-tuned on a synthetic dataset, up from 29.3% for the base model on the same held-out evaluation set. The paper behind the fine-tune introduces MobileActionsExtended, a set of about 9,500 records covering 15 device-control categories. UGrowAI released the weights, the datasets, the training code and a working Android app alongside it.
The 15 categories cover toggling lights, sending email and SMS, placing calls, reading battery and Bluetooth status, taking pictures and screenshots, adjusting screen brightness, and listing or opening applications. About one third of the prompts require more than one call. A second model trained jointly on the new data and Google’s own mobile actions dataset holds that extended score and reaches 82.3% on Google’s benchmark, which the authors describe as an 8 percentage point trade-off in return for doubling category coverage. The figures come from greedy decoding with no schema-constrained decoding, so the model has to emit a valid call on its own rather than being forced into the right format.
FunctionGemma 270M end-to-end accuracy on mobile function calling
| Benchmark | FunctionGemma 270M (fine-tuned) | Base FunctionGemma 270M |
|---|---|---|
| MobileActionsExtended | 76.48 | 29.33 |
| MobileActionsGoogle | 60.15 | 58.79 |
| MobileActionsExtended (combined model) | 76.48 | 29.33 |
| MobileActionsGoogle (combined model) | 82.31 | 58.79 |
Strict exact-match end-to-end accuracy in percent on the two held-out evaluation sets, all figures according to UGrowAI and measured on unquantized checkpoints. Google's own Mobile Actions specialist scores 90.32 on MobileActionsGoogle and 17.20 on MobileActionsExtended.
The demo app, Handy Chat, runs the model on a Google Pixel 8 Pro through the LiteRT-LM CPU backend rather than the GPU, with an exported bundle of 285.6 MB. The training data is synthetic rather than collected from real users, and the dataset and the demo cover only English and Android. Both fine-tuned models and both datasets are on Hugging Face under the UGrowAI account, with GitHub repositories for the fine-tuning pipeline and a downloadable APK.