Octopus v3 picks an action from an image and a query in under 1B parameters
Wei Chen and Zhiyuan Li posted Octopus v3 on April 17, 2024, a technical report on a multimodal agent model that takes an image alongside a text query and answers with a function call. The model has fewer than 1B parameters, handles English and Chinese in one set of weights, and is built to run on edge devices. The paper lists both authors at Stanford University and gives a Nexa AI address for correspondence.
The model encodes each supported function as a single functional token in its vocabulary, learned the way a language model learns an unfamiliar word from the contexts it appears in, so the output is one token for the action plus its arguments and needs no parser to read. Images run through a CLIP-based encoder, which the authors say they chose after comparing encoding methods. Training runs in stages, with the language model and the image encoder trained apart, then merged and aligned, then taught the functional tokens using the method from Octopus v2, and finally tuned by reinforcement learning with another large language model acting as the reward model.
The evaluation consists of 10 smartphone functions converted into functional tokens and ten worked examples, each printing the model’s output next to the output of GPT-4V feeding GPT-4, which the authors prompted with all the function descriptions and a few examples. The authors write that their model performs comparably to that pair on the selected functions. The report carries no accuracy rate, no success rate and no benchmark table, so the comparison rests on the ten printed generations rather than on a measured score.
The report states that the model runs on devices as constrained as a Raspberry Pi and that it makes converting a Raspberry Pi into something like a Rabbit R1 or a Humane AI Pin practical with an on-device model instead of a cloud service. It names no Raspberry Pi board, no phone and no other hardware it was run on, and gives no latency, no memory footprint and no throughput, so the edge deployment is asserted rather than shown. The authors do name one cost themselves, writing that vision input may introduce considerable latency and that they were optimising inference speed.
Weights and inference code were available only on application when the report went up, which describes the model as still under testing and for research purposes only, and the authors state that the functional token is licensed. The project page that hosted the demo videos no longer resolves. The authors list audio and video as the modalities they want to support next.