Three MoE Inference Optimizations NVIDIA Measured on DGX Spark

TL;DR

• The DGX Spark is a compact AI computer from NVIDIA for developers and researchers, built on the GB10 Grace Blackwell superchip with up to 128 GB of unified memory, and the model benchmarked on it here is Ling-3.0-flash from Inclusion AI, a mixture-of-experts model with 124 billion parameters.
• NVIDIA's post published on 27 October 2026 measures three changes to that model's decode loop at batch size 1, and reports time per output token falling from 27.71 ms to 9.72 ms.
• Profiling showed decoding was memory-bound: reading model weights drove execution time, and the MoE computation plus the output projection dominated it.
• Humming MXFP4 kernels compile at run time for the model's real shapes, which measured +9.3% decode throughput.
• Online FP8 quantization halves the LM head weight reads and brings the stack to +21.7% over baseline.
• Multi-token prediction with NEXTN turns one verification into four tokens, and the synthetic workload accepted 3.90 tokens per round.
• GSM8K measured a 42.5% throughput gain across 1,319 questions with accuracy unchanged within 0.08 of a point.
• The synthetic workload overstates the gain a real workload sees, because repetitive output is easier to draft.
• One missing interface makes the stack slower than baseline, which is why the vendor recommends validating each optimization at startup.

A batch of one is the least glamorous benchmark in AI infrastructure and the one most people actually run. It is the shape of a coding assistant answering one developer, a document summariser serving one request, an agent waiting on its own next token. NVIDIA chose that shape for this measurement, and the result is a useful case study in how much of a decode step is spent reading weights off memory.

Decoding was memory-bound on a 124-billion-parameter model at batch size 1

Ling-3.0-flash keeps 5.1 billion of its 124 billion parameters active per token. It has 42 layers, 35 of them Kimi Delta Attention linear-attention layers and seven multi-head latent attention layers, and each MoE layer routes a token to eight of 512 routed experts alongside one shared expert. The LM head projects a hidden dimension of 2,560 onto a vocabulary of 157,184 tokens.

At batch size 1, arithmetic units sit idle and the decode step waits on memory bandwidth. Two operations carried the cost: the routed-expert computation, and the vocabulary projection whose entire weight block has to be read for every generated token. NVIDIA's three optimizations attack those two costs in two different ways, and the article's value is that it reports each one separately before stacking them.

Humming MXFP4 kernels compile at run time for the model's real shapes

The baseline ran the official W4A8 recipe: FlashInfer MXFP4 MoE kernels, CUTLASS dense GEMM, and the --enable-fp32-lm-head option. Swapping the MoE backend for Humming moved time per output token from 27.71 ms to 25.35 ms and decode throughput from 36.09 to 39.45 tokens per second, a 9.3% gain.

The mechanism explains the modest size of the gain. Humming uses NVRTC to generate and compile CUBINs at run time for the actual M, N and K dimensions, the data types and the GPU architecture, then caches them in ~/.humming/cache. That specialisation matters on a model with a narrow moe_intermediate_size of 768, and it changes the build experience: an individual kernel compiles in seconds, while the FlashInfer CUTLASS path uses ninja to compile 97 targets and takes tens of minutes. A shorter build is a practical gain that never shows up in a throughput table.

Online FP8 quantization halves the LM head weight reads

The checkpoint stores the LM head in BF16 and lists it in modules_to_not_convert, so loading never quantizes it. Its vocabulary GEMM is 157,184 by 2,560, and every generated token reads that whole block. Under a bandwidth bottleneck it is the obvious target.

NVIDIA applies a dynamic FP8 quantization configuration when the LM head is constructed, leaving the rest of the model on the checkpoint's FP8 and MXFP4 settings. That halves the read volume for the layer, and the setting is one environment variable, SGLANG_ENABLE_FP8_LM_HEAD=1. The cumulative table shows 22.76 ms and 43.94 tokens per second, 21.7% over baseline and 11.4% over Humming alone. The flag is mutually exclusive with the baseline's FP32 logits option, which is worth knowing before someone enables both and wonders why nothing changed.

Multi-token prediction turns each verification into four tokens

The first two optimizations reduce the cost of a decode step. Speculative decoding reduces the number of steps, and Ling-3.0-flash ships a NEXTN prediction layer (num_nextn_predict_layers: 1) that works as a draft model with no extra training or download.

The configuration uses three draft steps and a four-token verification window: the draft predicts three tokens in sequence, and the target verifies four positions at once, accepting the correct prefix. The benefit depends on accept length and on the verification-cost ratio, which measured about 2.26. The interesting part of the analysis is where that cost came from. At batch size 1, small KV reads and larger weight reads made expert activation the principal verification cost: a normal decode activates eight experts for one token while verification over four tokens can activate up to 32.

On the synthetic 32K-input, 2K-output workload the stack reached 9.72 ms and 102.88 tokens per second, 185.1% over baseline, with an accept length of 3.90. That number is where a careful reader slows down.

GSM8K measured a 42.5% throughput gain across 1,319 questions

The second evaluation is the one worth trusting more. NVIDIA ran all 1,319 GSM8K test questions with five-shot prompting, temperature 0.6, top_p 0.95, top_k 29, a 32,768-token ceiling and chain-of-thought enabled, holding sampling, parser, seed and memory allocation fixed.

Across that set, time per output token went from 26.45 ms to 18.56 ms, decode throughput from 37.81 to 53.89 tokens per second, a 42.5% gain, and mean latency from 15.21 s to 11.49 s. P50 fell from 8.36 s to 6.11 s and P95 from 50.20 s to 38.60 s. GSM8K accuracy moved from 96.89% to 96.82%, and the run recorded zero errors and zero truncations. The measured gain is real and roughly a quarter of the synthetic claim.

The synthetic workload overstates the gain a real workload sees

NVIDIA states the caveat itself, which is rarer than it should be. Accept length was 3.90 on the synthetic run against 2.68 on GSM8K. Three differences explain most of the gap: synthetic output repeats, which makes drafting easy; the synthetic test decoded greedily while the real evaluation sampled at temperature 0.6 with top_p 0.95 and top_k 29; and the synthetic workload fixed output at 2K tokens while the median GSM8K output is about 310 tokens, which spreads the fixed overhead reductions over a much shorter generation.

That last point is the one to carry into a deployment decision. Speculative decoding pays per token generated, and the two other optimizations pay per step, so a workload with short answers collects most of the first two gains and a fraction of the third. The same discipline applies to any vendor benchmark: the useful question is which of the measured mechanisms your own traffic can reach. A previous analysis of how CUDA 13.4 turns a GPU fraction into an enforceable memory limit followed the same logic, because a capability only counts once the platform enforces it under your workload.

One missing interface makes the stack slower than baseline

Stacking the three optimizations is where the article earns its keep. Each worked alone. Together, the FP8 LM-head path and MTP need the draft model to share the entire LM-head module, because the draft has to interpret the same quantized weights as the target. The original Ling NEXTN implementation exposed only the weight tensor.

Without the interface the framework falls back silently: the draft receives FP8 weights and reads them as BF16, accept length collapses to 1.00, and throughput ends up below the configuration with no optimizations at all. Nothing fails, nothing logs an error, and the only symptom is that the machine is slower than it was before someone made it faster. NVIDIA's guidance is to validate at startup that every optimization is active, which is more reliable than inferring configuration state from a later performance number.

This paragraph belongs in an internal wiki. A silent fallback in an inference stack can create a capacity incident on launch day, so startup validation belongs in the deployment checklist. The model, the kernels and the quantization are NVIDIA's engineering; the discipline of proving each layer is active belongs to whoever operates the deployment. Related resources include Netics, and the deployment planning page.

Sources

• 3 Ways to Boost MoE Inference on NVIDIA DGX Spark, Yang Liu, developer.nvidia.com, 2026-10-27
• DGX Spark User Guide, docs.nvidia.com
• NVIDIA DGX Spark product page, nvidia.com

Source: the NVIDIA technical blog post of 2026-10-27, the DGX Spark user guide landing page and the official DGX Spark product page, all read on 2026-10-29. The Ling-3.0-flash layer counts, the expert routing, the 2,560 by 157,184 LM head, the 27.71 ms and 36.09 tokens per second baseline, the Humming and FP8 cumulative table figures, the 2.26 verification-cost ratio, the 3.90 and 2.68 accept lengths, every GSM8K figure including the 42.5% throughput gain and the 96.82% accuracy, the startup-validation guidance and the silent fallback behaviour all come from those pages.