Apple's MobileCLIP-S0 encodes an image in 1.5 ms on an iPhone 12 Pro Max

Five Apple researchers presented MobileCLIP at CVPR 2024 on June 20, a family of image-text models built for phones. They exported each model with Core ML Tools 7.0 and timed it at batch size 1 on an iPhone 12 Pro Max running iOS 17.0.3, and their Table 7 gives the smallest variant, MobileCLIP-S0, 1.5 ms for the image encoder and 1.6 ms for the text encoder at 67.8% zero-shot accuracy on ImageNet. The same authors measure OpenAI’s ViT-B/16 CLIP on that phone at 11.5 ms and 3.3 ms for 68.3%, and both models average 58.1 across the 38 datasets of the DataComp evaluation suite.

Multi-modal reinforced training, the method the paper introduces, moves the expensive part of distillation out of the training loop and into the dataset. Apple reinforces the DataComp dataset once, writing five synthetic captions per image with the CoCa model in OpenCLIP, storing the parameters of strong random image augmentations, 30 per image for the 12M subset and 10 for the billion-sample set, and storing the embeddings that two ViT-L/14 teacher models produce for those augmented images and for the real and synthetic captions. Training afterwards reads the captions and the teacher embeddings from disk, so the captioning model and the teachers never run again however many models are trained on the set.

The paper puts numbers on what that saves. One epoch over the 12.8M samples of the reinforced set on a single node of 8 A100-80GB GPUs takes 1.3 hours, the same as an epoch on the unreinforced data, against 21.1 hours when the captioning model and the teacher ensemble run during training, according to Table 4d. The cost moves to storage, and Table 4c puts DataComp-1B at 90 TB against 140 TB for the reinforced DataCompDR-1B, with the 12M subset going from 0.9 TB to 1.9 TB.

MobileCLIP comes in four sizes, and Table 7 lists image and text encoder parameters separately. S0 pairs an 11.4M-parameter image encoder with a 42.4M-parameter text encoder, S1 uses 21.5M and 63.4M at 2.5 ms and 3.3 ms, S2 uses 35.7M and 63.4M at 3.6 ms and 3.3 ms, and MobileCLIP-B keeps a standard ViT-B/16 image encoder at 86.3M and 10.4 ms. Zero-shot ImageNet accuracy across the four runs 67.8%, 72.6%, 74.4% and 76.8%, and a version of B trained on 39B seen samples instead of 13B reaches 77.2%. All three small variants replace the vision transformer with a convolution-transformer hybrid derived from Apple’s FastViT, and S0 also swaps the text encoder for one that mixes 1-D convolutions with four self-attention layers, which the authors report as 5% smaller and 15.8% faster than the 12-layer transformer text encoder that S1 and S2 keep.

Apple released the code, the weights and the reinforced data at apple/ml-mobileclip, with checkpoints for all four variants and the longer-trained B, and with DataCompDR-1B and DataCompDR-12M on Hugging Face as synthetic captions, augmentation parameters and teacher embeddings keyed to DataComp images rather than as the images themselves. The repository lists the code under MIT, the models under Apple’s own machine learning research model terms, and the data under CC BY-NC-ND. Apple showed zero-shot scene classification running in real time on an iPhone in the CVPR exhibit hall, and added an iOS app for the same task to the repository in November 2024.