Google's Gemma 4 MTP Drafters Now Deliver Up to 3x Faster Inference

If you've been running Gemma 4 locally and watching your tokens-per-second counter, Google just made your day significantly better. The team behind Gemma 4 has shipped Multi-Token Prediction (MTP) drafters for the entire Gemma 4 family, and the headline number is hard to ignore: up to 3x faster inference, with no compromise on output quality.

This isn't a model update. It's a decoding architecture change, and it targets one of the most stubborn bottlenecks in LLM deployment: the fact that standard autoregressive generation is painfully slow by design.

Gemma 4 had already surpassed 60 million downloads within weeks of launch. Stacking a 3x speed multiplier on top of that adoption curve is a meaningful move for the open-weight AI space.

The Problem MTP Solves

Standard LLM inference is memory-bandwidth bound, creating a significant latency bottleneck. The processor spends the majority of its time moving billions of parameters from VRAM to the compute units just to generate a single token, leading to under-utilized compute and high latency, especially on consumer-grade hardware.

This is the core inefficiency that has plagued local LLM deployment for years. Your GPU's compute cores sit idle while the memory bus ferries weights back and forth. Gemma 4 models generate text autoregressively and produce one token at a time, with roughly the same amount of compute needed for each token regardless of how difficult it is to predict. This makes it an unnecessarily slow process when the tokens are quite easy to predict.

MTP addresses this directly by changing how the model spends its idle compute time.

How Speculative Decoding with MTP Drafters Works

Speculative decoding decouples token generation from verification. By pairing a heavy target model (e.g., Gemma 4 31B) with a lightweight drafter (the MTP model), idle compute is used to "predict" several future tokens at once with the drafter in less time than it takes the target model to generate one.

If the target model agrees with the draft, it accepts the entire sequence in a single forward pass and even generates an additional token of its own in the process. This means your application can output the full drafted sequence plus one token in the time it usually takes to generate a single one.

The drafter itself is architecturally lean. It's described as a lightweight 4-layer MTP drafter that proposes candidate tokens, often referred to as the "assistant" since the model helps the larger model in choosing which tokens to predict.

When fewer tokens are drafted, the acceptance rate tends to be higher since tokens closer in position to the initial prompt are more accurate. However, since only a few tokens are drafted, the speedup from a faster drafter model is reduced. Hugging Face Transformers handles this tradeoff automatically via a heuristic schedule that adjusts the draft count at runtime based on acceptance rate.

Key Technical Highlights

To make the MTP drafters exceptionally fast and accurate, Google introduced several architectural enhancements. The draft models seamlessly utilize the target model's activations and share its KV cache, meaning they don't have to waste time recalculating context the larger model has already figured out.

Here's a breakdown of the three main architectural choices:

• Target Activations: The draft model uses the activations from the last layer of the target model, concatenates them with the token embeddings, and down-projects them to the drafter model's dimension.
• KV Cache Sharing: The draft model cross-attends to the target model's KV cache rather than building its own. This eliminates redundant context recomputation entirely.
• Efficient Embedder: For the E2B and E4B edge models, where the final logit calculation becomes a big bottleneck, Google implemented an efficient clustering technique in the embedder to further accelerate generation.

For the 26B mixture-of-experts model, while it presents unique routing challenges at a batch size of 1 on Apple Silicon, processing multiple requests simultaneously at batch sizes of 4 to 8 unlocks up to a ~2.2x speedup locally.

What Developers Actually Get

For developers, inference speed is often the primary bottleneck for production deployment. Whether you're building coding assistants, autonomous agents that require rapid multi-step planning, or responsive mobile applications running entirely on-device, every millisecond matters.

Pairing a Gemma 4 model with its MTP drafter delivers three concrete wins:

• Improved Responsiveness: Significantly lower latency for applications such as real-time chat, immersive voice systems, and agentic workflows.
• Supercharged Local Development: Enables the use of the 26B MoE and 31B Dense models on personal computers and consumer GPUs with remarkable speed, facilitating complex offline coding and workflows.
• Enhanced On-Device Performance: Maximizes the capabilities of E2B and E4B models on edge devices, generating outputs more quickly while preserving battery life, with zero quality degradation since the primary Gemma 4 model is responsible for final verification.

Availability and Tooling

The MTP drafters for the Gemma 4 family are available today under the same open-source Apache 2.0 license as Gemma 4. That licensing matters. Gemma 4 ships under Apache 2.0, the same license used by Qwen and most of the open-weight ecosystem, with no monthly active user caps, no acceptable-use policy enforcement, and full commercial freedom.

You can download the model weights right now on Hugging Face, Kaggle, and start experimenting with faster inference with Transformers, MLX, vLLM, SGLang, and Ollama, or try them directly on Google AI Edge Gallery for Android or iOS.

The Hacker News community has been quick to contextualize the speed gains. One commenter noted that it's not uncommon to see a Gemma vs Qwen comparison where Qwen does a bit better but spent 22 minutes on the task, while Gemma aligned the buttons wrong but only spent 4 minutes. "So taken at face value, Gemma is now underperforming leading open models by 5–10%, but doing it in 1/10th the time." Speed as a competitive axis is real, and this release leans into it hard.

Final Thoughts

The MTP drafter release is a well-executed inference optimization, not a model capability bump. What makes it technically interesting is the KV cache sharing design. Instead of running the drafter as a fully independent model, it piggybacks on the target model's already-computed context, which keeps the overhead low enough that the net gain is still a 3x multiplier in favorable conditions. That's a non-trivial engineering result.

The edge model story is also worth watching. Gemma 4 models are designed to deliver frontier-level performance at each size, targeting deployment scenarios from mobile and edge devices (E2B, E4B) to consumer GPUs and workstations (26B A4B, 31B). Combining that with MTP drafters means the E2B model running on a phone just got meaningfully faster without a hardware upgrade. For anyone building on-device AI products, that's a real change in what's feasible.

The open question I'd watch is acceptance rate variance across task types. Speculative decoding gains are highly dependent on how well the drafter's predictions align with the target model's distribution. Structured outputs, code, and repetitive text will see the biggest gains. Creative or highly unpredictable generation will see less. Google hasn't published per-task acceptance rate data yet, and that breakdown will matter for developers choosing whether to enable drafters by default.

What do you think? Are MTP drafters the right lever for local inference, or do you think the compute should go toward better quantization schemes instead? Drop your thoughts in the comments.

---