NVIDIA Answers HBM Pressure With Tiering: 3% More Memory, 30x More Throughput per Dollar
"NVIDIA's 5 November memory post pairs 3% more HBM per GPU with up to 30x higher throughput per dollar, and puts the rest of the gain in NVFP4 weights, KV cache offload to CMX storage and RA
TL;DR
- Full-stack memory codesign is a design practice at NVIDIA that treats GPU, CPU, networking, storage and software as one system so that each byte of memory delivers more finished inference work.
- The technical post of 5 November 2026 reports HBM capacity per GPU rising roughly 3% from Blackwell Ultra to the Vera Rubin platform, against up to 30x higher agentic throughput per dollar on a SemiAnalysis AgentX workload.
- HBM capacity defines the resident working set, and memory bandwidth governs its transfer rate; beyond the throughput plateau, extra capacity sits idle and continues to draw power and add cost.
- The wasted-byte fixes split three ways: NVFP4 weights spread across up to 72 GPUs, KV cache in NVFP4 and then offloaded to CMX storage through Dynamo, and RAG indexes that stay outside HBM.
- CMX is a storage tier a customer buys, sold through named partners, which turns context offload into a capacity and rack-cost decision.
- The settings the benchmark leaves to the operator are prefix-cache hit rate, tier placement and eviction policy, and none of the three appears in the published numbers.
- What stays open: one vendor-run benchmark on one model, with per-user token rates fixed at 100 and 125 tokens per second.
Three numbers from the benchmark suite
NVIDIA published the post on 5 November 2026 under Amr Elmeleegy's byline, and it makes a claim that reads oddly at first: memory grew 3% while throughput per dollar grew 30x. Both figures describe the same generational step. From the Blackwell Ultra platform to Vera Rubin, HBM capacity per GPU rose roughly 3%, and agentic throughput per dollar rose by up to 30x on SemiAnalysis AgentX.
The second number sits inside a longer comparison. On the SemiAnalysis AgentX DeepSeek V4 Pro workload, GB300 NVL72 reaches up to 10x the throughput per dollar of the H200 platform at 100 tokens per second per user, and there HBM per GPU roughly doubles. Vera Rubin NVL72 then reaches up to 30x against GB300 NVL72 at 125 tokens per second per user, and HBM per GPU rises about 3%.
Read as a sequence, the arithmetic says the generational gain stopped coming from capacity. Across the two steps, HBM growth moved from roughly 100% to about 3%. The post attributes the accompanying throughput gains to full-stack codesign. Something else moved the output, and the post spends most of its length naming that something.
The throughput plateau: HBM capacity and memory bandwidth impose separate limits
The post uses a warehouse comparison that holds up under inspection. HBM is the warehouse floor and memory bandwidth is the loading dock. A larger warehouse holds more inventory, and the dock still sets how much passes in and out per hour. Once the dock runs at capacity, extra shelving makes the building bigger without shortening a single delivery.
The engineering name for the ceiling is the memory wall. Every additional concurrent user shares the same compute and the same memory bandwidth, and at some point bandwidth stops supplying data as fast as the compute units consume it. Past that point an extra stack of HBM capacity cannot be put to work, and it still draws power and adds cost to the bill of materials. Across the fleet, that is the pattern the post calls the throughput plateau.
The chart in the post is the useful part of the argument, because it converts a marketing sentence into a decision. Capacity has a knee. Buy past the knee and the invoice grows while the token rate stays flat, which is a procurement failure disguised as a hardware upgrade.
NVFP4 weights spread across 72 GPUs
The first source of waste is weight precision. NVFP4 stores weights in four bits, and the post frames it against FP8, which it describes as the lowest precision available on the Hopper platform. Smaller weights leave more of each HBM stack for KV cache and for concurrent users, and the post lists models already shipping NVFP4 checkpoints: FLUX.2 [klein] from Black Forest Labs, Inkling from Thinking Machines Lab, Beam 1.0 from Reflection, Meta Muse Spark, Step 5 Preview from StepFun and NVIDIA Nemotron 3.5 Super VL.
Distribution handles the second half of the same problem. Inside an NVL72 system, weights can be spread across up to 72 GPUs, so each device holds a slice of the model with no full copy on any card. A model that would not fit on one accelerator becomes a resident workload across a rack, and the HBM freed on each device goes to context.
The interesting detail sits in the middle column. Precision work on weights is a well-trodden path. Applying NVFP4 to the KV cache as well, so more context stays resident in HBM, moves the same trick to the part of memory that grows with conversation length, and that part is what agentic traffic stresses.
KV cache in NVFP4, then offload to CMX through Dynamo
Agentic workloads change the shape of the problem. The post puts trajectories at up to 3 million tokens in some cases, with prior context carried across repeated model and tool calls. Each request keeps a working memory that grows while it runs, and retrieval-augmented generation adds enterprise passages to the same budget.
Two mechanisms take the pressure off. The Vera CPU shortens the time context sits idle during a tool call, because faster tool execution returns to the model sooner, and its LPDDR memory connects to Rubin GPUs over NVLink-C2C so paused context has somewhere near to wait. Dynamo then moves context that is genuinely idle into CMX context memory storage, a tier built for capacity at a much lower cost per bit than HBM, and brings it back before reuse. The post describes CMX as a shared tier across servers that omits long-term data-protection overhead for short-lived context.
Dynamo KV Cache Reuse is the piece that ties the tiers together, managing KV cache as one system-wide resource across GPU memory, CPU memory and storage. Context computed once can be reused across the system, an approach also examined in how inference throughput is actually measured : the number that matters is finished tokens per unit of power, and cache behaviour sets it more than peak FLOPS does.
A RAG index that stays outside HBM
The third fix targets retrieval. Nemotron Retriever models produce smaller embeddings with higher retrieval accuracy, and the cuVS library builds and serves the index across CPU and storage. The knowledge base then lives outside HBM, and only the passages a request needs enter the context window.
This is the least glamorous of the three and the one with the clearest operational consequence. Keeping an index on storage makes disk capacity the sizing constraint and reserves GPU memory for weights and live context.
CMX is a storage tier you buy
CMX arrives with a partner list: Cloudian, DDN, Dell Technologies, Everpure, HPE, IBM, NetApp, VAST and WEKA. A reader should take that list as the real content of the announcement. Context offload now carries a hardware line item, with capacity, latency and support terms attached.
The honest framing for a buyer is that the memory wall has a price list. Moving context to a shared tier reduces HBM per token, and it adds a dependency: the tier has to return a page before the model needs it, and the return path runs over a network. A slow tier turns an offload into a stall, and the stall arrives as added latency with no error message attached.
The three settings the benchmark leaves to the operator
The post publishes throughput per dollar at fixed per-user rates. Real fleets live on the three settings behind that number.
Prefix-cache hit rate decides how often a context is computed twice, and it depends on how requests are routed and how prompts are written. Tier placement decides which contexts sit in HBM, which wait in LPDDR, and which page out to CMX, and a poor policy can erase the savings from tiering. Eviction policy decides what leaves first under pressure, and a naive first-in-first-out rule can push out a context that is about to be reused.
What the numbers leave open
Three limits belong next to the 30x. The measurement is vendor-run, on one workload, with the model named (DeepSeek V4 Pro) and the per-user token rate fixed at 125. A different model, a different serving engine or a different arrival pattern moves the result. The comparison is generational, so part of the gain belongs to the rack, the fabric and the software, and the post attributes the whole of it to codesign. And the throughput figure is a ceiling, which is the nature of an up-to benchmark.
The practical reading of 5 November 2026 is that memory has become a placement problem spanning several operational tiers. A team that plans capacity from HBM per GPU will overspend on the plateau, and a team that plans from finished tokens per watt has to instrument prefix-cache behaviour first. The engineering work that survives a platform upgrade runs in the serving layer, which is the layer an operator controls, and it is where routing, tier policy and eviction rules live together with the monitoring that shows whether any of them pays. That layer is what an operator runs as managed cloud infrastructure, with the measurements attached to every change.
Sources
Source: the NVIDIA technical blog post of 5 November 2026 by Amr Elmeleegy, read in full, with its three in-body figures and campaign hero downloaded the same day. The 3% HBM growth, the 10x and 30x throughput-per-dollar figures, the 100 and 125 tokens per second per user measurement points, the 72-GPU distribution, the 3 million token trajectories, the NVFP4 checkpoint list, the CMX partner list and the Dynamo behaviour all come from that post; no number is carried from a secondary summary.