CUDA 13.4 Turns a GPU Fraction Into an Enforceable Memory Limit

TL;DR

• GPU fractioning is the practice of running several workloads on one physical GPU, each with its own share of memory and compute, so a small job stops consuming a whole device.
• NVIDIA released CUDA 13.4 on 27 October 2026 with a memory boundary the application cannot see, plus an open-source GPU Fractioning Operator that applies per-container shares on every node.
• The scheduler writes the contract into four Pod annotations, and any Kubernetes scheduler that can write those annotations can enforce GPU fractions with CUDA.
• KAI Scheduler v0.18.0 is the first integration, through a GPU-sharing mode called NvFractions that is enabled with one Helm flag and installs the operator as a dependency.
• Time-slicing gives each workload its own error domain and about 1/n of the time; SM-sharing runs everything at once on split streaming multiprocessors inside one shared error domain.
• The guaranteed memory request and optional burst limit are the two numbers to verify before a team enables this on a shared cluster.
• Memory enforcement settles the accounting question and leaves bandwidth, cache and failure domains exactly where they were.

Every platform team with a GPU node knows the arithmetic problem. As an illustration, a batch job that needs 8 GB of memory can hold a device with 80 GB while the other 72 GB sit idle because the pod asked for one GPU and the scheduler granted one GPU. NVIDIA's answer in CUDA 13.4 is to make the fraction real at the level where it can be enforced: the node.

CUDA 13.4 Makes the Memory Boundary Transparent to the Application

NVIDIA's GPUs already offered three ways to share a device. Multi-instance GPU splits a GPU into hardware-isolated instances that are transparent to the application. Time-slicing lets several processes take turns and enforces no memory limit. Multi-Process Service lets them run concurrently in separate address spaces, which NVIDIA calls sm-sharing.

CUDA 13.4 adds enforcement on top of that. The release enforces a memory boundary that is transparent to the application, in a new time-sharing mode, alongside the compute sharing mode that already existed in MPS. Four behaviours matter:

• An exact memory amount. A workload asks for a memory size and CUDA holds it there. An allocation above that size fails, exactly as it would on a smaller GPU.
• A range. The workload gets a guaranteed minimum, plus a maximum it can grow into when memory is free.
• A compute mode. Each workload selects its compute mode; time-slicing is the default, and sm-sharing is also available.
• A transparent limit. Each workload sees only a fraction of the GPU memory.

The failure mode is the interesting part. An allocation above the enforced amount fails, just as it would on a smaller GPU.

The Operator Runs MPS on Every Node and Applies What the Scheduler Wrote

CUDA provides the enforcement capability, and a Kubernetes cluster cannot reach it on its own. On every GPU node, something has to run and supervise the daemons these features depend on, apply each Pod container's memory limits and compute mode before the process starts, and report what each fraction actually uses. NVIDIA puts that work on the node, and says so directly: "This work belongs on the node, independent of whichever scheduler placed the workload."

The result is gpu-fractioning, an open-source Kubernetes operator hosted under the KAI Scheduler organisation, running next to the NVIDIA GPU Operator that supplies the driver and the container runtime setup. It has three jobs: set up and run MPS on every GPU node, apply each workload's memory limits and compute mode, and measure memory and compute use per fraction.

That split is the architectural decision worth studying. The scheduler chooses placement. The node enforces the limit. Two different programs, with a written contract between them.

Four Annotations Carry the Memory Contract

The contract is small enough to read in one screen. A scheduler records a workload's allocation and compute mode in per-container Pod annotations, and the operator reads them and configures the assigned GPUs.

NVIDIA states the general rule that follows: any Kubernetes scheduler that can write these annotations can enforce GPU fractions with CUDA. The scheduler keeps its own obligations, which are choosing a GPU with enough free memory and never mixing compute modes on one GPU.

Time-Slicing and SM-Sharing Divide Compute Two Ways

Compute can be divided as well, and call n the number of workloads sharing a GPU. Either way each workload gets roughly 1/n of the compute. The two modes schedule that share differently: time-slicing takes turns, SM-sharing runs workloads together.

Time-slicing is serial. Workloads take turns on the whole GPU, each gets about 1/n of the time when all of them use their full slice, and unused time is shared equally between the others. This is the default, and each workload runs in its own error domain, which is full error isolation.

SM-sharing is parallel. The streaming multiprocessors are split between workloads, which run concurrently while the hardware allocates and releases SMs as needed. When every workload is busy the split settles at roughly 1/n, and a single active workload can use them all. In this mode, workloads share a single error domain.

Read that last sentence twice before enabling anything. SM-sharing gives better utilisation and narrower blast protection. NVIDIA's own guidance is to use it for workloads you trust to run together. Each container declares a compute mode; KAI Scheduler tracks the single mode active on each shared GPU, and different GPUs on one node can use different modes.

A 40Gi Request on Four GPUs, and Half a GPU for a Notebook

NVIDIA ships two examples and they cover the two shapes most teams need.

The first is an inference server with a guaranteed 40Gi on each of four GPUs, a 60Gi limit it can grow into when neighbours are idle, sm-sharing, and gpu-fraction-num-devices: "4". The request applies to each device, so 40Gi means 40Gi on every one of the four.

The second is a development or batch job that asks for gpu-fraction: "0.5", a limit of gpu-fraction.limit: "0.8", time-slicing, and two devices. Half of each GPU guaranteed, growth to 80% when memory is free, and each job in its own error domain so a crash stays local.

One detail in the second example deserves attention. The limit is optional, and when it is omitted the limit equals the request, so the workload is capped at exactly what it asked for. A portion limit must be greater than the gpu-fraction and less than 1. Those constraints are what stops a team from writing a limit that silently re-creates the whole-device behaviour it was trying to remove.

What to Verify Before You Enable NvFractions

Run through this list on one node before it reaches a shared cluster.

• Confirm the NVIDIA GPU Operator is present, since the fractioning operator relies on it for the driver and the container runtime setup.
• Decide whether each workload needs an absolute size or a portion, because the limit annotation differs between the two request types.
• Write the burst limits on purpose. A generous limit is a performance decision with a neighbour cost attached.
• Monitor memory use for every fraction as its own accounting unit.
• Test whether nvidia-smi inside the container reports the fraction you expect, which is the sanity check NVIDIA uses in its own example.

Memory Enforcement Leaves Bandwidth and Failure Domains Alone

A fraction of memory is a fraction of one number. The bandwidth of the device, the L2 cache and the computational throughput remain shared, and time-slicing still serialises execution. A 40Gi memory allocation defines capacity; benchmark the workload separately to measure the performance of that share on the target GPU.

Hardware isolation remains the MIG story. MIG splits a GPU into instances that are isolated in hardware, and CUDA 13.4 adds enforcement to the sharing modes that sit below that line. Choosing between them is a question about tenants: a shared cluster of internal teams can live with a shared error domain, while a customer-facing one may want the hardware boundary and accept the coarser split.

The operational value sits in metering. Once a fraction is a declared, enforced quantity, a platform team can bill an internal team for 40Gi of a GPU the way it already bills for memory in a virtual machine. That is the change worth planning for, and it lands next to the broader work of building a sovereign infrastructure and cloud exit plan, where every shared resource needs an owner and a number.

NVIDIA states that any Kubernetes scheduler that can write the annotations can enforce GPU fractions with CUDA.

Two things are still open. NVIDIA documents the operator against KAI Scheduler and states that any scheduler able to write the annotations can use it, and it does not publish the per-fraction overhead of running MPS on every node. The second one is measurable on your own hardware before a rollout, which is the honest way to decide whether the density gain pays for the supervision.

Sources

Source: Memory-Enforced GPU Sharing in Kubernetes with NVIDIA CUDA 13.4 — developer.nvidia.com, 2026-10-27, by Ekin Karabulut, David Lifshitz and Hagay Sharon. All CUDA 13.4 behaviour, annotation names, example Pods and the KAI Scheduler v0.18.0 details come from that page (retrieved 2026-10-27).