SGLang Runs DeepSeek's Own Benchmarks Against DeepInfer and Matches It

TL;DR

• DeepInfer is the inference engine DeepSeek built and runs behind its own service, and DeepSeek open-sourced it together with the GPU kernels it runs on; a kernel is the small compiled routine that does the heavy arithmetic for attention, expert routing and matrix multiplication.
• The SGLang team published a measurement of its own engine against DeepInfer, running DeepInfer's own benchmark workload on the same hardware and retaining the disclosed differences in measurement methodology, and reports up to 1.04x prefill throughput on 4x GB300 and 4x B300 plus decode steps 1.1% to 7.4% shorter on 64x GB300, staying within 1% at the remaining points.
• DeepSeek's kernels, including FlashMLA, DeepGEMM, DeepEP, DeepSelect, the TileLang kernels and the Mega series, now run inside SGLang, along with DSpark speculative decoding, SWA bounded replay and both Engram placements.
• DeepInfer's README describes this release as offline inference and lists a KV cache system, prefill/decode disaggregation and load balancing across data-parallel replicas as requirements for a complete inference service. SGLang ships all three.
• The two engines were measured on the same hardware with DeepInfer's benchmark workload; the disclosed methods differ: DeepInfer times about nine prefill batches on GPU 0 and repeats the same prompt per GPU, and SGLang averages every complete batch on every GPU between 20% and 90% completion with different new tokens per request.
• What the results leave open for a serving team is the capacity question underneath the throughput number: where the KV cache tier lives, whether the draft model carries redundant experts, and what a gateway has to do when replicas hold different cached prefixes.

What DeepSeek published and the README's complete-service requirements

DeepInfer runs behind DeepSeek's own serving, and DeepSeek published it with its kernels. The repository describes it as a high-performance, large-scale inference framework for DeepSeek models. For anyone building on DeepSeek weights, that release sets the number to beat: it is the vendor measuring its own model on its own stack.

The README also carries a short section that matters more than its length. It names a KV cache system, VE/Prefill/Decode disaggregation, and load balancing across data-parallel replicas for both prefill and decode among the requirements for a complete inference service. Those three items are operational requirements for turning fast kernels into a continuously available service.

The DeepSeek kernels SGLang now runs

The integration list is long and specific. SGLang runs FlashMLA, DeepGEMM, DeepEP, DeepSelect and the TileLang kernels, together with the Mega series: MegaAttention, MegaMoE, MegaGate and Mega-mHC. MegaMoE fuses token dispatch, expert GEMMs and the combine into one kernel, and MegaAttention is the default attention path on B300 and GB300, on single servers and on both prefill and decode servers.

Three mechanisms operate above the kernels. DSpark speculative decoding runs the checkpoint's draft model to propose a block of tokens the target model verifies together, which cuts the number of target-model forward passes. SWA bounded replay runs the late decoder layers on the last 128 prompt tokens and the earlier source layers continue to process the full prompt, and on a prefix-cache hit, encoder replay rebuilds missing sliding-window KV from the last 128 cached tokens.

Engram placement gets both options. With SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 each node holds one copy of the tables in host memory, which leaves the most HBM for the KV cache. With SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS=1 the tables are split across GPUs and each GPU reads the rows it needs straight from its peers over NVLink, with no all-gather or reduce-scatter around the lookup.

How the two engines were measured

Both engines ran DeepInfer's benchmarks on the same hardware. Prefill used a 4-GPU node with DP4/EP4, 128K-token contexts and 4K new tokens per request, where each GPU prefills B new tokens per step from 4K to 64K, with balanced expert routing, EPLB off and 32 redundant experts. Decode used 64 GB300 GPUs across 16 nodes at DP64/EP64, 128K-token contexts with a shared cached prefix, three draft tokens per step with static verification, and 384 routed plus 64 redundant experts.

The measured results vary with the workload. On 4x GB300, SGLang leads at 4K, 32K and 64K new tokens per GPU and sits level at 8K and 16K. On 4x B300 it leads at 4K, 8K and 32K and sits level at 16K and 64K. On the decode side, with 64 GB300 GPUs at EP64, its step is 1.1% to 7.4% shorter from 16 to 256 requests per GPU and level at 8.

The post also explains where the difference comes from. Mega-mHC feeds MegaMoE directly, attention produces MXFP8 output for the output projection, and the prefill compressor and request-window KV copies use fused kernels, which reduces intermediate memory traffic and the number of kernel launches. Breakable CUDA graphs cover prefill steps of up to 8K tokens per GPU, including long contexts and data-parallel attention, and decoder replay uses separate graphs sized to the bounded suffix.

Where the two measurement methods differ

The appendix discloses four differences, and they belong next to the numbers. DeepInfer measures how long prefill takes for about nine batches on GPU 0, and SGLang averages every complete batch on every GPU between the points where 20% and 90% of requests have finished. DeepInfer repeats the same prompt across requests on each GPU, and SGLang uses different new tokens per request.

Configuration accounts for two of the disclosed differences. SGLang keeps the Engram tables on device and DeepInfer runs them in host memory, and DeepInfer's draft model carries 64 redundant experts and SGLang's draft carries none. Each choice affects memory pressure, verification cost and the resulting chart, so the post states both configurations explicitly.

The scheduling idea DeepInfer credits to SGLang

Since v0.4 in December 2024, SGLang has prepared step N+1 on the CPU during GPU execution of step N, overlapping scheduling with execution. The post notes that DeepInfer's documentation credits this design to SGLang, which makes the comparison partly a test of whether the older engine still holds up against a newer one that inherited parts of its design.

DeepSeek's chat format arrived as well. Prompts are rendered and outputs parsed with DeepSeek's own deepseek-recipe package, so reasoning and tool calls in SGLang's API follow DeepSeek's format, and grammar-constrained decoding covers JSON schema, regex and EBNF. A serving engine that speaks the model vendor's own format removes a translation layer from the deployment.

What the results leave open for a serving team

Production decisions center on the three complete-service requirements named in DeepInfer's README. SGLang's unified radix cache reuses KV across requests and conversation turns, and the configuration in the post uses Linker to connect L1 GPU caches to shared L3 storage such as 3FS or Mooncake, with HiCache offering a host-memory L2 tier. Prefill and decode run on separate servers with Mooncake, NIXL or MORI as the transfer backend, and the model gateway distributes requests across replicas with routing based on cached prefixes.

Where the KV cache tier lives is now a capacity decision with a bill attached. HBM, host memory and shared L3 storage trade latency against capacity and cost, and that trade sits inside any sovereign infrastructure and cloud exit review of an inference estate.

A team choosing between the two engines this month has a short list of questions for capacity planning. The capacity plan should identify which of the three missing components are already in operation and their cost. The traffic pattern should be compared with both measurement scopes: about nine batches on one GPU and a cluster average of every completed batch. The capacity decision should determine whether the prefix cache hit rate justifies a disaggregated deployment and how the draft model's expert configuration affects the memory budget.

Sources

Source: More Than DeepInfer: What SGLang Delivers — LMSYS Org, The SGLang Team — https://www.lmsys.org/blog/2026-10-24-dsv41-deepinfer/, published 2026-10-24, retrieved 2026-10-24 (DeepSeek open-sourced DeepInfer with the kernels it runs on; SGLang measured against it on DeepInfer's own benchmarks on the same hardware; up to 1.04x prefill throughput on 4x GB300 and 4x B300 with a lead at 4K, 32K and 64K on GB300 and at 4K, 8K and 32K on B300; decode steps 1.1% to 7.4% shorter on 64x GB300 from 16 to 256 requests per GPU; FlashMLA, DeepGEMM, DeepEP, DeepSelect, TileLang kernels and the Mega series MegaAttention, MegaMoE, MegaGate and Mega-mHC integrated; DSpark speculative decoding; SWA bounded replay on the last 128 prompt tokens with encoder replay from the last 128 cached tokens; Engram tables held in host memory with SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 or split across GPUs with SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS=1 and direct peer reads over NVLink; zero-overhead scheduling since v0.4 in December 2024, credited by DeepInfer's documentation to SGLang; DeepSeek chat format through the deepseek-recipe package with JSON schema, regex and EBNF constrained decoding; benchmark commits 7d996300b9 for prefill and 3b95120f00 for decode on the SGLang side and 774ce24 for DeepInfer; prefill on a 4-GPU node at DP4/EP4 with 128K-token contexts, 4K new tokens, balanced expert routing, EPLB off and 32 redundant experts; decode on 64 GB300 GPUs across 16 nodes at DP64/EP64 with three draft tokens per step, static verification and 384 routed plus 64 redundant experts, step time converted at 3.25 accepted tokens per step; disclosed differences where DeepInfer times about nine batches on GPU 0 and repeats the same prompt per GPU, SGLang averages every complete batch on every GPU between 20% and 90% completion with different new tokens per request, SGLang keeps Engram tables on device and DeepInfer uses host memory, and DeepInfer's draft model carries 64 redundant experts and SGLang's carries none).

Source: deepseek-ai/DeepInfer repository README — GitHub — https://github.com/deepseek-ai/DeepInfer, retrieved 2026-10-24 (DeepInfer described as a high-performance, large-scale inference framework for DeepSeek models; components required for a complete inference service and listed as not included in the release: a KV cache system, VE/Prefill/Decode disaggregation, and load balancing across data-parallel replicas for both prefill and decode).

Source: More Than DeepInfer, What SGLang Delivers — The SGLang Team, lmsys.org, published 24 October 2026, retrieved 2026-10-24, with the deepseek-ai/DeepInfer README from github.com. Figures reproduced from the SGLang post; the two comparison visuals are Netics editorial diagrams built from the post and the README.