Last week, NVIDIA published a technical post about running mixture-of-experts models faster on DGX Spark. The engineers chose Ant’s Ling-3.0-flash for the test and worked with the Ant Ling team on the deployment recipe.
The headline number was a 42.5% increase in decode throughput.
Ling went from 37.81 to 53.89 tokens per second on the same DGX Spark. Mean response latency also dropped from 15.21 to 11.49 seconds. Accuracy on the GSM8K math benchmark stayed almost identical, moving from 96.89% to 96.82%.
You can read the comprehensive research and test results here:
3 Ways To Boost Moe Inference On Nvidia Dgx SparkNVIDIA changed three parts of the inference setup:
- The kernel that runs Ling’s experts
- The precision of its output layer
- The way it predicts upcoming tokens
The result gives local AI developers a practical example of how much speed can still be found after a model already fits on the hardware.
Ling has enough capacity for coding, research, and agent workflows, yet only a small part of the model runs for each token. Ant Ling describes the idea as getting “more useful work per token.” NVIDIA’s experiment shows what that looks like once the model meets a serving stack tuned for its architecture.
This is an incredibly important piece of information because it shows how small these models can get while still maintaining strong reasoning and coding capabilities. I’ve kept an eye on nearly every coding model that can run locally, including Qwen3.8–27B, DeepSeek-V4-Flash, GLM-5.3-Flash, and many other open-weight models.
So, it made me wonder why NVIDIA’s researchers chose Ling-3.0-flash for this experiment.
Why NVIDIA Chose Ling-3.0-Flash
NVIDIA’s 42.5% result came from a close match between Ling-3.0-flash, the GB10 chip inside DGX Spark, and several changes in SGLang. The model choice explains a lot of what happened next.
Take note that even though Ling-3.0-flash behaves like a much smaller model during inference, its 124 billion parameters are still crazy big on disk and in memory.
I checked the model’s download files on Hugging Face, and the total size is around 255 GB. Ant’s design gets around the compute problem through a mixture-of-experts, or MoE, architecture.
The Ling model card lists 512 routed experts. Every token passes through eight of those experts and one shared expert. That leaves only 5.1 billion active parameters per token, a small fraction of the full model.
This setup gives Ling access to a large pool of learned weights while keeping each generation step relatively light.
The entire expert pool still has to sit somewhere in memory, though. Sparse activation reduces the amount of computation, while quantization and DGX Spark’s unified memory handle the physical size of the checkpoint.
Ling also comes with features that suit long agent runs. It has a native 256K context window, reasoning that can be turned on or off, and training aimed at coding, tool calls, deep research, and multi-step tasks.
Its built-in NEXTN layer can predict upcoming tokens, which later became the largest source of acceleration in NVIDIA’s test.
NVIDIA did not say it compared Ling with other models, but my guess is that it offered a better test for DGX Spark.
Ling gave the engineers three clear targets in one model:
- Narrow MoE layers
- A huge vocabulary layer
- A built-in NEXTN speculative decoding
Its FP4 checkpoint also fits within DGX Spark’s unified memory. A smaller dense model would have been easier to run, but it would not have tested as many parts of the inference stack.
DGX Spark Can Hold the Whole Model
NVIDIA DGX Spark is a compact desktop AI system built around the GB10 Grace Blackwell Superchip. The 128 GB version has enough unified memory to hold models that would overwhelm the VRAM on most consumer GPUs.
This is what the actual chip looks like:
The official DGX Spark page lists up to 1 petaFLOP of FP4 AI performance, a 20-core Grace CPU, and 128 GB of unified system memory. NVIDIA rates that configuration for inference with models up to 200 billion parameters.
Ant Ling’s optimized checkpoint is much smaller than the full-precision model. The deployment notebook puts the MXFP4 weights at roughly 60 GB.
Step 3: Download Ling-3.0-flash FP4 (MXFP4) Model Weights
Pre-quantized MXFP4 weights (~60GB) are hosted on:
ModelScope Ling-3.0-flash-fp4
Hugging Face Ling-3.0-flash-fp4
Download model weights to ~/models/Ling-3.0-flash-fp4 using ModelScope:
!source .venv/bin/activate && uv pip install -U modelscope
!source .venv/bin/activate && uv run modelscope download --model inclusionAI/Ling-3.0-flash-fp4 --local-dir ~/models/Ling-3.0-flash-fp4
That leaves room for the server, caches, compiled kernels, and other memory needed during generation.
The shared memory pool is the reason this 124B model can run on a single box. There’s no need to split its experts across several GPUs or send traffic through an interconnect.
For someone building a private coding assistant or local research agent, that makes the whole setup easier to manage.
It also creates a different performance problem. DGX Spark can hold the weights, but it still has to move them quickly. During single-request decoding, NVIDIA found that Ling spent much of its time reading weights from memory. Extra compute power won’t help much while the processor is waiting for those reads to finish.
Where Ling Was Losing Time and What NVIDIA Changed
NVIDIA tested Ling-3.0-flash at batch size one, similar to how someone might run a local assistant. The profiling showed that decoding was limited mainly by memory bandwidth. Most of the delay came from repeatedly reading model weights.
Two parts were especially expensive. The first was Ling’s MoE layer, which routes every token through eight experts and one shared expert. The second was its LM head, which scores outputs across a vocabulary of 157,184 tokens.
Ling’s built-in NEXTN layer offered another opportunity. It can draft several upcoming tokens and let the main model verify them together, reducing the number of decoding rounds.
NVIDIA turned these findings into three optimizations:
- Humming kernels tuned for Ling’s expert shapes
- FP8 quantization for the large LM head
- NEXTN speculative decoding to generate several tokens per round
The first two made each decoding step cheaper. The third reduced the number of steps needed to generate an answer. Together, they produced the 42.5% throughput improvement.
Let’s discuss them one by one.
1. A Better Kernel for Ling’s Experts
The baseline ran Ling’s experts through the FlashInfer MXFP4 backend. NVIDIA replaced it with Humming, a kernel library that compiles GPU code around the model’s exact dimensions, data types, and hardware.
Press enter or click to view image in full size
This reduced the time per output token from 27.71 milliseconds to 25.35 milliseconds, raising throughput by 9.3%.
This optimization fits local inference especially well. A personal workstation often runs the same checkpoint repeatedly, which makes a one-time compilation step easy to justify.
The model does not become smaller or smarter. The software simply becomes better at running its unusual expert shapes.
It is also a reminder that model format is only part of local performance. Two machines can load the same quantized weights and still produce different speeds because their inference kernels are different.
2. Compressing the Output Head
Ling’s output layer, or LM head, maps 2,560 hidden values across a vocabulary of 157,184 tokens. Its weights were stored in BF16 and had to be read for every generated token.
NVIDIA converted this layer to FP8 during loading, cutting its weight traffic roughly in half. Together with Humming, throughput reached 43.94 tokens per second, a 21.7% improvement over the original setup.
Here’s an illustration to make things easier to understand.
One thing developers can easily miss is that quantization is not always applied evenly across a model. Ling’s main weights were already compressed, while the LM head remained at higher precision. Targeting that one layer produced a noticeable gain without rebuilding the entire checkpoint.
This also shows how vocabulary size affects inference. A larger vocabulary may help a model cover more languages, code syntax, and special tokens, but the final projection becomes heavier.
3. Predicting Several Tokens at Once
Ling already contains this prediction layer, so the setup doesn’t require a second draft model or another large checkpoint. That keeps the memory footprint reasonable on a single DGX Spark.
Remove upgrade prompts from your unlocked stories in your settings.
Checking those predictions still costs compute because one normal decode step activates eight routed experts. Verifying four positions can activate up to 32 experts, and NVIDIA measured each verification round at about 2.26 times the cost of standard decoding. The draft has to predict enough correct tokens to recover that extra work.
On a synthetic 32K-input and 2K-output test, NEXTN accepted an average of 3.90 tokens per round. The complete stack reached 102.88 tokens per second, 185.1% above the baseline.
That huge number didn’t carry over to normal prompts. Synthetic output tends to repeat itself, which makes the next few tokens easier to guess. The full GSM8K run gave NVIDIA a more sensible estimate.
What the 42.5% Result Covers
NVIDIA ran the optimized setup across all 1,319 questions in GSM8K. Throughput increased from 37.81 to 53.89 tokens per second, while mean latency dropped from 15.21 to 11.49 seconds. Accuracy stayed almost identical at 96.82%.
The published results were strong without looking suspiciously perfect:
- Decode throughput: 37.81 tokens per second increased to 53.89, up 42.5%.
- Time per output token: 26.45 milliseconds dropped to 18.56, a 29.8% reduction.
- Mean latency: 15.21 seconds dropped to 11.49, a 24.5% reduction.
- P95 latency: 50.20 seconds dropped to 38.60.
- GSM8K accuracy: 96.89% changed to 96.82%. NVIDIA found no statistically detectable accuracy difference.
- Time to first token: 0.207 seconds changed to 0.214 seconds.
The time to first token barely changed, moving from 0.207 to 0.214 seconds. NVIDIA made generation faster, not prompt processing.
A long coding response should benefit more than a short chat reply, and retrieval or tool calls will still take the same amount of time. A 42.5% increase in decoding speed does not mean an agent will finish its entire task 42.5% sooner.
