Gemma 2 2B is distilled from a larger model and scores 1126 on Chatbot Arena

Google released Gemma 2 2B on July 31, 2024, an open model of 2.6B parameters that Google says produces outsized results by learning from larger models through distillation. Google reports it at 1126 Elo on the LMSYS Chatbot Arena leaderboard, captured on July 30, 2024, ahead of Mixtral 8x7B Instruct at 1114, GPT 3.5 Turbo 0314 at 1106 and Llama 2 70B chat at 1093, and states that the model surpasses all GPT-3.5 models there.

The technical report describes training the 2B size on 2 trillion tokens with knowledge distillation, where a bigger teacher model supplies a probability for every token in context and the small student is trained to match those probabilities rather than to predict the next token. In an ablation the authors ran, a 2B model trained over 500B tokens distilled from a 7B teacher averaged 67.7 across three benchmarks, against 60.3 for the same model trained from scratch. The released model holds 2.02B non-embedding parameters plus 590M embedding parameters, which come from the 256k-entry Gemini vocabulary, and it interleaves local sliding-window attention with global attention over an 8192-token context. Google DeepMind trained it on 512 TPUv5e chips.

Google names edge devices and laptops as deployment targets next to cloud serving on Vertex AI and Google Kubernetes Engine, and says the model is small enough for the free tier of T4 GPUs in Google Colab. For running it locally Google points to Ollama and to Gemma.cpp, its own lightweight C++ inference engine, and says support in MediaPipe, the on-device inference stack for Android, iOS and the web, is coming. Google adds that NVIDIA optimised the model with the TensorRT-LLM library for GeForce RTX cards and for Jetson modules at the edge.

Google put the weights on Kaggle, Hugging Face and Vertex AI Model Garden under the Gemma terms, which it calls commercially friendly for research and commercial applications, and made the model available to try in Google AI Studio. The same announcement added ShieldGemma, a set of safety classifiers built on Gemma 2 that filter hate speech, harassment, sexually explicit and dangerous content, and Gemma Scope, over 400 sparse autoencoders for inspecting the inner workings of Gemma 2 2B and 9B.