How the scheduler sees a GPU

Kubernetes knows about two resources: CPU and memory. Everything else — GPUs, FPGAs, RDMA NICs — is an extended resource, which is a polite way of saying “an integer the kubelet has been told about”.

This post is about how that integer gets there, what the scheduler does with it, and where it falls short.

The device plugin handshake

A GPU node runs a small daemon, the device plugin, that registers with the kubelet over a Unix socket and advertises a list of device IDs. The kubelet turns that list into a capacity on the node:

Capacity:
  cpu:                64
  memory:             515Gi
  nvidia.com/gpu:     4
Allocatable:
  cpu:                63500m
  memory:             500Gi
  nvidia.com/gpu:     4

From here on the scheduler treats nvidia.com/gpu exactly like memory: a number to subtract from. A pod asks for some, the node has enough or it does not.

Requesting a GPU

The request lives in the container spec alongside CPU and memory. Note the highlighted lines — GPUs must be requested as limits, and requests and limits must be equal; you cannot overcommit an extended resource.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama-3-8b
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: vllm
          image: ghcr.io/example/vllm:0.9.1
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: 48Gi
            requests:
              cpu: '8'
              memory: 48Gi
      nodeSelector:
        nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB

When the pod is scheduled, the device plugin’s Allocate call hands back environment variables and device mounts (/dev/nvidia0, the driver libraries) so the container sees exactly one card.

What the integer does not know

Here is the part that bit us. The scheduler counts devices; it knows nothing about them.

Each gap has a workaround, and each workaround is a project:

• Model selection — labels from the GPU feature discovery daemon and a nodeSelector (see the YAML above).
• Topology — the topology-manager policy on the kubelet plus a scheduler plugin that understands NVLink domains.
• Gang scheduling — a batch scheduler (Kueue, Volcano) that admits all-or-nothing, so a 4-node training job never sits half-scheduled holding GPUs hostage.
• Fractional GPUs — the subject of part 3.

A quick sanity check

Before trusting any dashboard, I run this on a node and compare it with what the scheduler thinks:

nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total --format=csv

If the scheduler says the node is fully allocated and nvidia-smi says utilisation is 11 %, the platform is paying for four GPUs and using half of one. That gap — allocated versus busy — is the single most useful number a GPU platform can put on a graph, and the integer the scheduler works with cannot see it.