Hagasu marks scrollable regions on screen so mobile GUI agents find hidden targets

Hagasu is a vision-language model that reads a mobile screenshot and marks the areas a user can scroll, so a GUI agent knows when to swipe instead of tapping whatever is already in view. A team at Fudan University reports in a paper on arXiv that the model finds 78.46% of the scrollable regions in its test set, measured as recall against human annotations. Used as a plug-in, it raised the mean completion rate of three agents on 18 tasks that require exploration from 64.81% to 82.72%.
Clicking works far better than swiping in the authors’ measurements. Across seven GUI agents in a screenshot-based planning study, they report a weighted mean success of 90.1% for clicks on visible targets against 42.9% for the swipes needed to reveal hidden ones, a gap of 47.3 percentage points. In 83.73% of the failed swipe tasks, the agent clicked an incorrect visible target instead of scrolling.
Completion rate on 18 swipe-dependent tasks, in percent
| Benchmark | With Hagasu | Without |
|---|---|---|
| Qwen3.8-27B | 88.89 | 70.37 |
| Qwen3.7-Plus | 88.89 | 74.07 |
| GPT-6-Luna | 70.37 | 50.00 |
| Mean | 82.72 | 64.81 |
All figures according to the paper's authors, measured on a Xiaomi 13 running Android 16.
Training used 667 manually annotated screenshots, split into 540 for training, 60 for validation and 67 for testing, with a two-stage LoRA fine-tune of Qwen3.5-9B at rank 16 that adapts the vision side first and the language side second. At run time an annotation service predicts the region boxes for each decision screenshot, draws them onto the image and adds a text cue, which leaves the agent’s own model untouched. The paper puts the cost of that step at 49.46 seconds of extra execution time per run on average, ranging from 33.40 to 71.07 seconds across agents.

The model separates a horizontally scrollable filter row from the vertical result list on the same screen.Hu et al.
Tasks ran on a Xiaomi 13 with Android 16, but the paper does not say where Hagasu itself runs and gives no quantization, memory or latency figures for the model on a phone. The source code, the swipe dataset and the experimental records are on figshare, with no license stated for the code. The abstract describes the end-to-end result as covering nine GUI agents, while the body reports it for the three models selected for online evaluation.