Why GPUs change the bottleneck

I have spent most of my working life on Kubernetes, observability and the plumbing between them. When the first GPU node pool landed in our cluster I asked the obvious question:

Why can’t AI workloads run like normal microservices?

At first glance they are the same shape. Both are packaged into containers. Both run on Kubernetes. Both expose an API. Both scale on demand. So why is there an entire ecosystem growing around “AI infrastructure”?

After a few weeks of breaking things, the answer turned out to be simple: it is all about where the bottleneck lives.

A traditional request

Most services follow a familiar pattern.

user → api → database

A request reaches an API pod, some business logic runs, a row is fetched from a database, and a response goes back. The things we watch are CPU, memory, database latency, network round trips, and how many replicas we need to keep p99 under the SLO.

Platform teams have two decades of tooling for this. Horizontal scaling is cheap because pods are cheap. An idle CPU for a few minutes is nothing to worry about.

An inference request

An inference request looks almost the same on the wire, and completely different underneath.

user → inference server → gpu → model weights

Instead of querying a database, the process has to hold a multi-gigabyte model in GPU memory and push every token through it. The bottleneck stops being I/O and becomes:

• GPU memory (does the model even fit?)
• GPU utilisation (is the card actually busy, or waiting on the CPU?)
• model loading time (minutes, not milliseconds)
• memory bandwidth between host and device
• network throughput between nodes for anything multi-GPU

The challenge shifts from serving data efficiently to serving compute efficiently.

GPUs are expensive, so idle is a bug

One insight reframed everything for me.

In traditional infrastructure, an idle CPU is an acceptable cost of headroom. On a GPU node, idle is a bug you can measure in dollars per hour.

A single accelerator can cost more per hour than the rest of the node combined. That single fact changes what “good” looks like for a platform:

1. Bin-packing matters more than spreading. The default scheduler likes to spread pods for resilience. With GPUs you often want the opposite: pack the cards you have before you pay for another.
2. Cold starts are the enemy. Pulling a 15 GB image and loading weights takes minutes. Scale-to-zero, the darling of serverless, quietly becomes scale-to-a-cold-start.
3. Utilisation is the KPI. Not request rate, not replica count. If the GPU sits at 20 % busy, the platform is wasting most of what it pays for.

What this series covers

This is the first of three posts written while building a GPU platform for a few dozen product teams. The next two get concrete:

• Part 2 looks at how Kubernetes actually sees a GPU — device plugins, extended resources, and why nvidia.com/gpu: 1 is a much blunter instrument than it looks.
• Part 3 is about sharing a GPU between workloads without regretting it: time-slicing, MIG and MPS, and how to choose.

If you only take one thing from this post: AI workloads are not special because of the model. They are special because the scarce resource moved, and the whole platform has to move with it.