If you've been running Gemma 4 locally and watching your tokens-per-second counter, Google just made your day significantly better. The team behind Gemma 4 has shipped Multi-Token Prediction (MTP) drafters for the entire Gemma 4 family, and the headline number is hard to ignore: up to 3x faster inference, with no compromise on output quality.
This isn't a model update. It's a decoding architecture change, and it targets one of the most stubborn bottlenecks in LLM deployment: the fact that standard autoregressive generation is painfully slow by design.
Gemma 4 had already surpassed 60 million downloads within weeks of launch. Stacking a 3x speed multiplier on top of that adoption curve is a meaningful move for the open-weight AI space.
The Problem MTP Solves
Standard LLM inference is memory-bandwidth bound, creating a significant latency bottleneck. The processor spends the majority of its time moving billions of parameters from VRAM to the compute units just to generate a single token, leading to under-utilized compute and high latency, especially on consumer-grade hardware.
This is the core inefficiency that has plagued local LLM deployment for years. Your GPU's compute cores sit idle while the memory bus ferries weights back and forth. Gemma 4 models generate text autoregressively and produce one token at a time, with roughly the same amount of compute needed for each token regardless of how difficult it is to predict. This makes it an unnecessarily slow process when the tokens are quite easy to predict.
MTP addresses this directly by changing how the model spends its idle compute time.
How Speculative Decoding with MTP Drafters Works
Speculative decoding decouples token generation from verification. By pairing a heavy target model (e.g., Gemma 4 31B) with a lightweight drafter (the MTP model), idle compute is used to "predict" several future tokens at once with the drafter in less time than it takes the target model to generate one.
If the target model agrees with the draft, it accepts the entire sequence in a single forward pass and even generates an additional token of its own in the process. This means your application can output the full drafted sequence plus one token in the time it usually takes to generate a single one.
The drafter itself is architecturally lean. It's described as a lightweight 4-layer MTP drafter that proposes candidate tokens, often referred to as the "assistant" since the model helps the larger model in choosing which tokens to predict.
When fewer tokens are drafted, the acceptance rate tends to be higher since tokens closer in position to the initial prompt are more accurate. However, since only a few tokens are drafted, the speedup from a faster drafter model is reduced. Hugging Face Transformers handles this tradeoff automatically via a heuristic schedule that adjusts the draft count at runtime based on acceptance rate.
