Microsoft trains phi-1 to 50.6 percent on HumanEval with 1.3B parameters

Microsoft Research published phi-1 on June 20, 2023, a Python code model with 1.3B parameters. The paper reports 50.6 percent pass@1 on HumanEval and 55.5 percent on MBPP, pass@1 being the share of problems the model solves with its first generated answer. The same table lists StarCoder at 15.5B parameters and 1 trillion training tokens with 33.6 percent and 52.7 percent, GPT-3.5 at 175B parameters with 47 percent, and CodeGen-Mono at 16.1B parameters and 577B tokens with 29.3 percent.

The training data comes to under 7B tokens in three parts. About 6B tokens are Python files pulled from The Stack and StackOverflow by a classifier the authors trained on roughly 100,000 GPT-4 ratings of how instructive a snippet is for someone learning to code. Under 1B tokens are synthetic Python textbooks written by GPT-3.5, and about 180M tokens are synthetic exercises with solutions, used only in the finetuning step that turns phi-1-base into phi-1.

Microsoft ran the training on 8 Nvidia A100 GPUs, reaching the base model in under 4 days and spending another 7 hours on the finetuning. That amounts to about 8 passes over the pretraining data and a little over 50B tokens seen, which the paper’s chart puts at 1090 GPU hours for the 1.3B run.

The same pipeline at 350M parameters yields phi-1-small, which the authors put at 45 percent on HumanEval and use to argue that parameter count still drives which capabilities appear at all. Microsoft posted phi-1 on Hugging Face in September 2023, where the model card lists the MIT license.