TinyLLaVA's 3.1B model outscores 7B LLaVA-1.5 on seven of nine benchmarks

Researchers at Beihang University and Tsinghua University posted TinyLLaVA on February 22, 2024, a framework for building small vision-language models out of an existing language model and an existing image encoder. Their best configuration, which the paper calls TinyLLaVA-3.1B, scores higher than the 7B LLaVA-1.5 on seven of the nine benchmarks in the paper’s comparison table, ties it on GQA at 62.0, and trails it on MME, where the same table gives LLaVA-1.5 1510.7 against 1464.9. The authors report 79.9 against 78.5 on VQAv2 and 75.8 against 63.4 on LLaVA-Bench-in-the-Wild, and put the model ahead of the 7B Qwen-VL on four of the five benchmarks they share, behind it on TextVQA at 59.1 against 63.8.

The framework treats four things as variables, the small language model, the vision encoder that turns an image into patch features, the connector that maps those features into the language model’s embedding space, and the training recipe. The authors instantiate it with TinyLlama at 1.1B, StableLM-2 at 1.6B and Phi-2 at 2.7B as language models, CLIP at 0.3B and SigLIP at 0.4B as vision encoders, and either the 558k-caption LLaVA-1.5 data or the 1246k-caption ShareGPT4V data. Their base recipe updates only the connector during pre-training, while their share recipe freezes the first 12 layers of the vision encoder and updates the rest of the model.

The ablations put most of the movement in the vision encoder and the recipe. Swapping CLIP for SigLIP raises the Phi-2 variant’s TextVQA score from 51.4 to 55.6 and the TinyLlama variant’s from 45.8 to 49.1, which the authors attribute in part to SigLIP running at 384 pixels against 336 and emitting 729 visual tokens against 576. A two-layer MLP connector beat a resampler on four of six benchmarks in the single comparison the paper runs, 74.0 against 70.1 on VQAv2 and 58.1 against 54.2 on GQA, with the resampler ahead on SQA-I and MM-Vet. The authors report that the share recipe, which fine-tunes part of the vision encoder rather than freezing all of it, improved every variant pre-trained on ShareGPT4V.

Model size still helps when data and recipe are held fixed, and with CLIP under the base recipe the Phi-2 variant leads the StableLM-2 and TinyLlama ones on all seven benchmarks in the appendix table. The ordering flips once the recipe changes, with the 1.5B SigLIP TinyLlama model on the share recipe beating the 1.9B CLIP StableLM-2 model on the base recipe on six of seven benchmarks. The authors add a hallucination caveat, reporting that the share recipe lowers POPE accuracy for the StableLM-2 and Phi-2 variants even as their other scores rise, and they conjecture that smaller language models need more trainable parameters during pre-training while larger ones suffer from them.

The paper carries no on-device measurement. It states no latency, no memory footprint and no target hardware, and argues the small-scale case from training and inference cost rather than from phone or edge deployment. The authors wrote that weights and code would be made public, and the project repository and the 3.1B weights carry the Apache 2.0 license.