Benchmarks
62 updates on Benchmarks.
Hugging Face trains SmolLM2 at 135M, 360M and 1.7B on up to 11T tokens
Hugging Face released SmolLM2 in three sizes trained on up to 11 trillion tokens, with 4-bit builds from 118 MB for on-device runtimes.
Apple team finds H100 last on tokens per dollar for models up to 2B
Seven Apple authors measured tokens per dollar for LLaMA-style models from 100M to 2B and found H100s last at every size, behind cheaper A100s.
Mistral puts Ministral 3B and 8B on devices with 128k context
Ministral 3B and 8B handle up to 128k tokens for on-device work, but only the 8B Instruct weights were published, and for research use.
PalmBench finds iPhones running local LLMs about three times faster than Android phones
A benchmark of quantised LLMs on eight phones and boards reports throughput, memory, power and heat, with iPhones ahead and 4-bit drawing more power than 3-bit.
Gemma 2 2B is distilled from a larger model and scores 1126 on Chatbot Arena
Google released a 2.6B-parameter Gemma 2 trained by distilling a larger model, and reported an Elo of 1126 on the LMSYS Chatbot Arena.
Spectra ships ternary 3.9B models that match 4-bit quantisation at half the bits
Nolano AI trained 54 models from 99M to 3.9B parameters on the same 300B tokens in ternary, quantised and half-precision form to compare them by bit size.
Hugging Face releases SmolLM at 135M, 360M and 1.7B parameters
Three base models trained on the newly released SmolLM-Corpus, with published memory footprints from 109.78 MB to 3422.76 MB.
Apple's MobileCLIP-S0 encodes an image in 1.5 ms on an iPhone 12 Pro Max
Apple timed its image-text models on an iPhone and released four variants, the weights and the reinforced DataCompDR dataset.
TensorOpera releases Fox-1, a 1.6B model trained on 3 trillion tokens
TensorOpera published a 1.6B model under Apache 2.0 and reports it ahead of Gemma-2B and Qwen1.5-1.8B on a six-benchmark average.
BUPT measures 22 LLMs on four Android phones at about 200 ms per token
A BUPT team ran 22 models from 0.5B to 7B on four Android phones with llama.cpp and found 7B models take about 4 GB and 200 ms per token.
BUPT proposes one 9.2B model in the OS that all apps call through adapters
A BUPT team proposes that the phone OS ship one 9.2B multimodal model all apps call through small adapters, and matched app models on 85% of 50 datasets.
Apple releases OpenELM at 270M to 3B, with parameters spread unevenly across layers
Apple published four models from 270M to 3B parameters with the full training framework and code to run them through MLX on Apple silicon.