Enforcing GPU Memory Fractions in Kubernetes with CUDA 13.4 and KAI Scheduler

Giving every AI workload a whole GPU can leave capacity idle, and sharing a GPU only works if each workload stays inside its share. The NVIDIA Technical Blog post describes a path from a Pod specification to enforcement on each GPU node. The blog lists five steps, expanded in the sections below:

1. The developer writes resource and compute-mode annotations in the Pod YAML.
2. KAI Scheduler, in its NvFractions mode, selects the GPUs and writes per-container Pod annotations.
3. GPU Fractioning Operator on the node reads those annotations for the GPUs assigned to each container.
4. The operator applies each container's memory limits and compute mode before the process starts.
5. The operator measures memory and compute use per fraction rather than per whole GPU.

How GPU Fractioning Operator Enforces Fractions on Each Node #

CUDA provides memory enforcement, but a Kubernetes cluster cannot use it on its own. The blog says something on every GPU node must run and supervise the daemons these features depend on, apply each container's memory limits and compute mode, and report what each fraction uses. It states that this work belongs on the node, independent of whichever scheduler placed the workload.

GPU Fractioning Operator is the open-source Kubernetes operator that does this work. It is hosted under the KAI Scheduler organization on GitHub and runs next to the NVIDIA GPU Operator, which provides the driver and container runtime setup. The blog lists three jobs for it: setting up and running Multi-Process Service (MPS) on every GPU node, applying each workload's memory limits and compute mode, and measuring memory and compute use per fraction.

CUDA 13.4 Adds Memory Enforcement With a New Time-Sharing Mode #

CUDA 13.4 enforces a memory boundary that is transparent to the application. The blog says this comes with a new time-sharing mode, alongside the sm-sharing compute mode that already existed in MPS. Its intro describes time-slicing as letting processes take turns without memory enforcement, so the new mode is the one that adds the enforcement.

The blog lists four capabilities from the release. A workload can request an exact memory amount, and an allocation above that size fails. It can request a range with a guaranteed minimum and a maximum it can grow into when memory is free. It can choose a compute mode, and it sees only a fraction of the GPU memory. Each capability needs the node-side operator to configure it for the workload.

KAI Scheduler v0.18.0 Introduces the NvFractions Mode #

KAI Scheduler v0.18.0 adds the NvFractions GPU-sharing mode. The blog calls KAI Scheduler the first scheduler integrated with gpu-fractioning. The scheduler picks which slice of which GPU each workload gets, writes the annotations, and leaves enforcement to the operator on the node.

The blog states that KAI Scheduler's other sharing modes remain available and that administrators choose one per cluster. It also notes that any Kubernetes scheduler able to write these annotations can enforce GPU fractions with CUDA. That scheduler still has to choose a GPU with enough free memory and never mix compute modes on one GPU.

Per-Container Pod Annotations That Carry Each Fraction #

Each container receives its own memory, compute-mode and device annotations. The blog's Table 1 lists the keys the operator reads. Because the container name is part of each key, a request targets the right container in a multi-container Pod.

The scheduler writes these keys after it selects the GPUs, so the operator applies the scheduler's placement without making the placement itself. Engineers who write their own scheduler integration should set the same four keys, since the blog presents them as the contract the operator reads.

Memory Requests as Absolute Sizes or Portions of a GPU #

A request can be an absolute size or a portion of one GPU. The absolute form uses the per-container NVIDIA annotation with standard Kubernetes units, such as "40Gi". The portion form uses KAI Scheduler's existing gpu-fraction annotation, with a value such as "0.5", which the blog's example applies as half a GPU.

Choosing the form also determines which limit key applies. The blog's Table 2 pairs an absolute request with nvidia.com/container.<c>.gpu-memory.limit, and a portion request with kai.scheduler/container.<c>.gpu-memory.portion.limit. Teams that mix both forms in one cluster therefore need to check which key each Pod uses.

Burst Limits and the Portion Limit Key #

If no limit is set, the limit equals the request. The blog describes the limit as optional. Without it, the workload is capped at exactly what it requested. With a higher limit, the workload keeps its guaranteed minimum and can grow into free memory.

For portion requests, the blog states that the portion limit must be greater than the gpu-fraction and less than 1. Its burst example writes the key as gpu-fraction.limit with the value "0.8", while Table 2 uses the kai.scheduler portion key. The sources do not say which spelling applies to which release, so confirm the key against your installed KAI Scheduler before relying on it.

Time-Slicing and SM-Sharing Compute Modes per GPU #

Compute mode is set per container, and one GPU runs one mode at a time. The blog gives the choices as time-slicing, the default, and sm-sharing. KAI Scheduler never places a time-slicing workload and an sm-sharing workload on the same GPU, and it tracks this per GPU rather than per node, so one node can host both modes on different GPUs.

The blog explains that time-slicing is serial and gives each workload its own error domain, while sm-sharing is parallel and puts workloads in a single error domain. Under sustained load, each workload gets roughly 1/n of the compute, where n is the number of workloads sharing the GPU. This suggests that teams should choose sm-sharing only for workloads they trust to fail together.

metadata:
  annotations:
    nvidia.com/container.app.gpu-memory.request: "40Gi"
    nvidia.com/container.app.gpu-memory.limit: "60Gi"       # optional
    nvidia.com/container.app.gpu-compute.mode: "sm-sharing" # optional, default time-slicing
    gpu-fraction-num-devices: "4"                           # optional, default 1
spec:
  schedulerName: kai-scheduler

This is the blog's first example, an inference server. The blog describes it as needing a guaranteed 40Gi on each of four GPUs, growing to 60Gi when neighbors are idle, and running alongside other workloads. These are the intended behaviors the annotations request, and the post does not report measured results for them.

Spreading a Fraction Across Several GPUs #

The memory request applies to each device, not to the total. The gpu-fraction-num-devices annotation sets how many GPUs the workload spans, and the blog's example says a 40Gi request with four devices receives 40Gi on each of four GPUs.

The request applies to each device.

NVIDIA Technical Blog

Capacity planning therefore multiplies the per-device request by the device count to estimate total GPU memory. The blog's second example uses gpu-fraction "0.5", gpu-fraction.limit "0.8" and two devices with time-slicing, for a development or batch job that needs half of each of two GPUs.

metadata:
  annotations:
    gpu-fraction: "0.5"
    gpu-fraction.limit: "0.8"  # optional
    nvidia.com/container.app.gpu-compute.mode: "time-slicing"  # optional
    gpu-fraction-num-devices: "2"  # optional
spec:
  schedulerName: kai-scheduler

Enabling NvFractions and Finding the Guides #

NvFractions is enabled with a Helm flag during KAI Scheduler installation. The blog gives the command as helm upgrade -i kai-scheduler with the --set global.nvFractions.set=true flag, and it notes that this also installs gpu-fractioning as a dependency. Administrators should expect the operator to arrive with the scheduler, not as a separate step.

The blog points to the GPU Fractioning Operator repository for installation and configuration, and to the admin and user guides for all three sharing mechanisms. It invites questions through a Slack channel and bi-weekly community meetings, and mentions KubeCon + CloudNativeCon NA 2026 in Salt Lake City, Utah, on Nov. 9-12.