Open weights
42 updates on Open weights.
Microsoft carries its textbook data recipe from code to reasoning with the 1.3B phi-1.5
The 1.3-billion-parameter model trains on 30B tokens of mostly synthetic data and posts reasoning scores above Llama2-7B in Microsoft evaluations.
Microsoft trains phi-1 to 50.6 percent on HumanEval with 1.3B parameters
Microsoft Research reports 50.6 percent on HumanEval and 55.5 percent on MBPP for a 1.3B code model trained on under 7B tokens of textbook-quality data.
RWKV trains like a transformer and runs with constant memory per token
The paper releases pretrained RNN weights from 169M to 14B parameters and reports constant time and memory per token during inference.
LaMini-LM distils models from 61M parameters up on 2.58M instructions
MBZUAI distilled 17 instruction models from 61M to 7B parameters on 2.58M synthetic instructions, and reports the 248M model matching LLaMA-7B.
Hugging Face's DistilBERT is 40% smaller than BERT and 71% faster on an iPhone
The 66M-parameter distilled BERT keeps 97% of its teacher on GLUE and weighs 207 MB inside a question-answering app on an iPhone 7 Plus.
TinyBERT shrinks BERT-base to 14.5M parameters and runs 9.4x faster
Huawei's distilled BERT keeps more than 96.8% of its teacher's GLUE score at 14.5M parameters, 7.5x smaller and 9.4x faster on inference.