https://llmobile.news/tags/nexa-ai/A concise news ticker covering AI on mobile devices, local models, apps and hardware.en-GBhttps://llmobile.news/ticker/octopus-v3/https://llmobile.news/ticker/octopus-v3/Wed, 17 Apr 2024 17:07:06 +0200Wei Chen and Zhiyuan Li posted Octopus v3 on April 17, 2024, a technical report on a multimodal agent model that takes an image alongside a text query and answers with a function call. The model has fewer than 1B parameters, handles English and Chinese in one set of weights, and is built to run on edge devices. The paper lists both authors at Stanford University and gives a Nexa AI address for correspondence. The model encodes each supported function as a single functional token in its vocabulary, learned the way a language model learns an unfamiliar word from the contexts it appears in, so the output is one token for the action plus its arguments and needs no parser to read. Images run through a CLIP-based encoder, which the authors say they chose after comparing encoding methods. Training runs in stages, with the language model and the image encoder trained apart, then merged and aligned, then taught the functional tokens using the method from Octopus v2, and finally tuned by reinforcement learning with another large language model acting as the reward model. The evaluation consists of 10 smartphone functions converted into functional tokens and ten worked examples, each printing the model’s output next to the output of GPT-4V feeding GPT-4, which the authors prompted with all the function descriptions and a few examples. The authors write that their model performs comparably to that pair on the selected functions. The report carries no accuracy rate, no success rate and no benchmark table, so the comparison rests on the ten printed generations rather than on a measured score. The report states that the model runs on devices as constrained as a Raspberry Pi and that it makes converting a Raspberry Pi into something like a Rabbit R1 or a Humane AI Pin practical with an on-device model instead of a cloud service. It names no Raspberry Pi board, no phone and no other hardware it was run on, and gives no latency, no memory footprint and no throughput, so the edge deployment is asserted rather than shown. The authors do name one cost themselves, writing that vision input may introduce considerable latency and that they were optimising inference speed. Weights and inference code were available only on application when the report went up, which describes the model as still under testing and for research purposes only, and the authors state that the functional token is licensed. The project page that hosted the demo videos no longer resolves. The authors list audio and video as the modalities they want to support next. Source: https://arxiv.org/abs/2404.11459 Read the article: https://llmobile.news/ticker/octopus-v3/Nexa AIAgentsResearchhttps://llmobile.news/ticker/octopus-v2/https://llmobile.news/ticker/octopus-v2/Tue, 02 Apr 2024 11:01:00 +0200Wei Chen and Zhiyuan Li published Octopus v2 on April 2, 2024, a 2B function-calling model fine-tuned from Google’s Gemma 2B that turns a spoken or typed request into an Android API call. On their own evaluation set built from 20 Android APIs, the authors report 99.524% accuracy and 0.38 seconds per call for Octopus v2, against 98.571% and 1.02 seconds for GPT-4. Both authors give Stanford University as their affiliation on the paper and publish the weights under the name Nexa AI. The distinguishing idea is what the authors call a functional token. Each of the 20 APIs gets its own special token added to the tokenizer and to the model’s output head, so picking a function becomes a single choice among 20 token candidates instead of writing the function name out as text and hoping every piece of it lands correctly. Since the model learned during fine-tuning what each of those tokens stands for, the function descriptions no longer have to sit in the prompt, which the paper puts at over 95% less context. Retrieval-augmented function calling processes roughly 1,000 tokens per call in their comparison, while Octopus v2 sees the query alone, typically under 30 tokens. Shorter context is also where the speed comes from. The authors measure Llama-7B with retrieval-augmented function calling at 68.095% accuracy and 13.46 seconds per call on a single NVIDIA A100, and Octopus v2 at 0.38 seconds under the same settings on the same A100, a 35-fold difference. The phone figure is a separate measurement. With the model quantised and the state for the fixed prompt prefix precomputed, they report a function call finishing in 1.1 to 1.7 seconds for queries of 20 to 30 tokens on what the paper calls a standard Android phone, without naming the device, and they estimate 37 times more calls per iPhone battery charge than a 7B model with retrieval. The authors name limits of their own result. Training with LoRA instead of the full model drops accuracy to 99.048%, and cutting the training set from 1,000 to 100 generated examples per API drops it to 98.095%, which they still put forward as the low end of a usable range. The set of callable functions is fixed at training time, since every API has to exist as a token before the model can emit it, and they price the data generation at 0.0224 USD per 1,000 examples per API. They also note that the GPT-3.5 and GPT-4 latencies were taken through OpenAI’s API on March 18, 2024, and may reflect API traffic or hardware allocation at that moment. Nexa AI released the weights on Hugging Face under a non-commercial CC BY-NC 4.0 license, on top of Google’s Gemma 2B. The model card said at the time that the team was still preparing the code for an open-source release. Source: https://arxiv.org/abs/2404.01744 Read the article: https://llmobile.news/ticker/octopus-v2/Nexa AIAgentsAndroidGemmaResearchhttps://llmobile.news/ticker/octopus-function-calling/https://llmobile.news/ticker/octopus-function-calling/Tue, 02 Apr 2024 03:29:28 +0200Wei Chen, Zhiyuan Li and Mingyuan Ma of Stanford University and Harvard University posted Octopus on April 2, 2024, four open models fine-tuned to pick the right software API for a request and fill in its arguments. On a benchmark the authors built themselves, they report their fine-tuned Gemma 2B answering 93 percent of calls correctly against 96 percent for GPT-4 and 50 percent for GPT-3.5, with the CodeLlama 7B and Gemma 7B versions reaching 97 percent once an output filter is switched on. The title says on-device, but nothing in the paper runs on a device. The authors fine-tuned the models on A100 80GB GPUs and queried GPT-4 and GPT-3.5 through OpenAI’s API, so every number was measured server-side. The device claim is a single sentence stating that models of this size can already be deployed on mobile, pointing at MLC LLM, and the paper names no phone, no chip and no latency, memory or throughput figure. The training set starts from roughly 30,000 of the most used APIs on RapidAPI Hub, which GPT-4 cut to about 20,000 by dropping entries with missing arguments or descriptions that did not match their parameters. The authors rewrote each API as a Python function signature with a docstring, had GPT-4 write five answerable queries per API and an equal share of unanswerable ones, and let GPT-4 verify the result, ending at about 150,000 examples as described in the paper. Vector search over the function descriptions pulled in near-identical functions as distractors, and training worked through them from easy to hard. Base models were CodeLlama 7B, Gemma 2B and 7B and Stable Code 3B, all tuned with LoRA at rank 16 and 8-bit quantisation, using 90 GPU hours for CodeLlama 7B and Gemma 7B, 60 for Stable Code 3B and 30 for Gemma 2B. The output filter the authors call a conditional mask blocks, at each generated token, anything that would break the expected format, so a function name has to come from the known list and an argument has to match its declared type. The authors state that it cannot be applied to GPT-4 or GPT-3.5 because the OpenAI API does not return the underlying token scores, and that GPT-3.5 was not counted wrong for formatting mistakes. The paper states that the training dataset and the fine-tuned models will be open sourced soon, and names no repository or date. The preprint went up under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 license. Source: https://arxiv.org/abs/2404.01549 Read the article: https://llmobile.news/ticker/octopus-function-calling/Nexa AIResearchAgentsGemmaBenchmarks