Google adds multi-token prediction to Gemini Nano v3 on Pixel 9 and 10

Google Research has retrofitted multi-token prediction (MTP) onto the production Gemini Nano v3 models, which Google says recently rolled out to the Pixel 9 and Pixel 10 series. A small Transformer head attached to the last layers of the model proposes several future tokens per pass, and the unchanged main model verifies them in parallel. According to Google, MTP correctly predicts an average of nearly two additional tokens per inference pass in AI Notification Summaries and Proofread.

Gemini Nano v3 stays frozen and only the MTP head is trained, so Google says the output is bit-for-bit identical to the original model and its capabilities and safety alignment do not change. The head cross-attends to the main model’s existing key-value cache instead of keeping its own, which avoids a second prefill pass. Google reports savings of 130 MB per instance compared with a standalone drafter, a separate small model that proposes tokens in classic speculative decoding, because the head needs no embedding lookup tables, prefill attention variants or application-specific tuning parameters.

Google measured speedups of 50% or more on Pixel 9 depending on the task, compared with standalone drafters of comparable parameter count. For highly predictable outputs such as smart replies, token acceptance improved by up to 55%. The company says features like notification summaries, Proofread and smart replies now generate text faster and with less energy, and that developers no longer need to fine-tune a separate drafting model for each new task.

Google plans to bring MTP to future Pixel devices. The research team also wants to explore parallel decoding without auxiliary heads to cut draft latency, models that explore several branches in parallel, and a relaxed verification step that no longer requires an exact token match for some use cases.