Alibaba builds Qwen2's 0.5B and 1.5B sizes for phones, earphones and glasses

Alibaba’s Qwen Team published the Qwen2 Technical Report on July 15, 2024, covering five models from 0.5B to 72B parameters. The report states that the two smallest, Qwen2-0.5B and Qwen2-1.5B, are designed for easy deployment on portable devices such as smartphones, earphones and smart glasses, while the larger sizes target GPUs. Qwen2-1.5B scores 56.5 on MMLU, a multiple-choice knowledge test, against 52.7 for Microsoft’s Phi-2, which the same table credits with about twice the non-embedding parameters.

Both small sizes tie the input embedding to the output layer, so a single weight matrix does both jobs and the shared vocabulary of 151,646 tokens is stored once rather than twice. The report puts non-embedding parameters at 0.3B for Qwen2-0.5B and 1.2B for Qwen2-1.5B. All five models use grouped query attention, where several query heads share one set of key and value heads, with 14 query heads over 2 key-value heads at 0.5B and 12 over 2 at 1.5B, which the authors say lowers the key-value cache size per token against Qwen1.5 and with it the memory footprint during long-context inference.

Alibaba trained Qwen2-0.5B on a 12 trillion token dataset and the other dense models on a 7 trillion token one, reporting that the larger corpus did not pay off at the bigger sizes. For the instruction-tuned versions the report gives Qwen2-1.5B-Instruct 52.4 on MMLU, 61.6 on the GSM8K grade-school maths set and 29.0 on IFEval, which checks whether a model obeys explicit formatting instructions, against 37.9, 40.1 and 20.0 for Qwen2-0.5B-Instruct. The authors attribute the gain over the similarly sized Qwen1.5 models mainly to that data scaling.

Context length differs across the family. The authors tested retrieval from long documents with Needle in a Haystack and report that the two small models handle 32K tokens, Qwen2-57B-A14B-Instruct 64K, and the 7B and 72B instruction-tuned models 128K with the YARN extension applied. The pre-training data covers approximately 30 languages, among them English, Chinese, Spanish, French, German, Arabic, Russian, Korean, Japanese, Thai and Vietnamese.

Alibaba put the weights on Hugging Face and ModelScope. The Qwen2 blog post of June 7, 2024 states that Qwen2-0.5B, 1.5B, 7B and 57B-A14B carry the Apache 2.0 license while Qwen2-72B and its instruction-tuned variant stay on Alibaba’s own Qianwen License, and it names MLX, llama.cpp, Ollama and LM Studio among the projects that run the models locally.