Sam Ortega

Writing with MDX: callouts, tabs, and snippets

hello@example.com (Sam Ortega) — Sun, 05 Jul 2026 00:00:00 GMT

A tour of the authoring components this theme ships with — how to use them, and when plain Markdown is the better choice.

Sharing a GPU without regret

hello@example.com (Sam Ortega) — Fri, 05 Jun 2026 00:00:00 GMT

Time-slicing, MIG and MPS solve the same problem three different ways. Here is how to pick, and the failure mode of each.

How the scheduler sees a GPU

hello@example.com (Sam Ortega) — Wed, 03 Jun 2026 00:00:00 GMT

Kubernetes knows about two resources: CPU and memory. Everything else — GPUs, FPGAs, RDMA NICs — is an extended resource, which is a polite way of saying “an integer the kubelet has been told about”.

This post is about how that integer gets there, what the scheduler does with it, and where it falls short.

The device plugin handshake

A GPU node runs a small daemon, the device plugin, that registers with the kubelet over a Unix socket and advertises a list of device IDs. The kubelet turns that list into a capacity on the node:

Capacity:
  cpu:                64
  memory:             515Gi
  nvidia.com/gpu:     4
Allocatable:
  cpu:                63500m
  memory:             500Gi
  nvidia.com/gpu:     4

From here on the scheduler treats nvidia.com/gpu exactly like memory: a number to subtract from. A pod asks for some, the node has enough or it does not.

Requesting a GPU

The request lives in the container spec alongside CPU and memory. Note the highlighted lines — GPUs must be requested as limits, and requests and limits must be equal; you cannot overcommit an extended resource.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama-3-8b
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: vllm
          image: ghcr.io/example/vllm:0.9.1
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: 48Gi
            requests:
              cpu: '8'
              memory: 48Gi
      nodeSelector:
        nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB

When the pod is scheduled, the device plugin’s Allocate call hands back environment variables and device mounts (/dev/nvidia0, the driver libraries) so the container sees exactly one card.

What the integer does not know

Here is the part that bit us. The scheduler counts devices; it knows nothing about them.

The scheduler knows	The scheduler does not know
how many GPUs	which model (A100 vs L4) unless you label nodes
how many are free	how much GPU memory is free on a shared card
node affinity rules	whether two GPUs share an NVLink or cross the PCIe
pod priority	that your job holds 8 GPUs across 4 nodes as a unit

Each gap has a workaround, and each workaround is a project:

Model selection — labels from the GPU feature discovery daemon and a nodeSelector (see the YAML above).
Topology — the topology-manager policy on the kubelet plus a scheduler plugin that understands NVLink domains.
Gang scheduling — a batch scheduler (Kueue, Volcano) that admits all-or-nothing, so a 4-node training job never sits half-scheduled holding GPUs hostage.
Fractional GPUs — the subject of part 3.

A quick sanity check

Before trusting any dashboard, I run this on a node and compare it with what the scheduler thinks:

nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total --format=csv

If the scheduler says the node is fully allocated and nvidia-smi says utilisation is 11 %, the platform is paying for four GPUs and using half of one. That gap — allocated versus busy — is the single most useful number a GPU platform can put on a graph, and the integer the scheduler works with cannot see it.

Why GPUs change the bottleneck

hello@example.com (Sam Ortega) — Mon, 01 Jun 2026 00:00:00 GMT

I have spent most of my working life on Kubernetes, observability and the plumbing between them. When the first GPU node pool landed in our cluster I asked the obvious question:

Why can’t AI workloads run like normal microservices?

At first glance they are the same shape. Both are packaged into containers. Both run on Kubernetes. Both expose an API. Both scale on demand. So why is there an entire ecosystem growing around “AI infrastructure”?

After a few weeks of breaking things, the answer turned out to be simple: it is all about where the bottleneck lives.

A traditional request

Most services follow a familiar pattern.

user → api → database

A request reaches an API pod, some business logic runs, a row is fetched from a database, and a response goes back. The things we watch are CPU, memory, database latency, network round trips, and how many replicas we need to keep p99 under the SLO.

Platform teams have two decades of tooling for this. Horizontal scaling is cheap because pods are cheap. An idle CPU for a few minutes is nothing to worry about.

An inference request

An inference request looks almost the same on the wire, and completely different underneath.

user → inference server → gpu → model weights

Instead of querying a database, the process has to hold a multi-gigabyte model in GPU memory and push every token through it. The bottleneck stops being I/O and becomes:

GPU memory (does the model even fit?)
GPU utilisation (is the card actually busy, or waiting on the CPU?)
model loading time (minutes, not milliseconds)
memory bandwidth between host and device
network throughput between nodes for anything multi-GPU

The scarce resource moves from the database to the accelerator.

The challenge shifts from serving data efficiently to serving compute efficiently.

GPUs are expensive, so idle is a bug

One insight reframed everything for me.

In traditional infrastructure, an idle CPU is an acceptable cost of headroom. On a GPU node, idle is a bug you can measure in dollars per hour.

A single accelerator can cost more per hour than the rest of the node combined. That single fact changes what “good” looks like for a platform:

Bin-packing matters more than spreading. The default scheduler likes to spread pods for resilience. With GPUs you often want the opposite: pack the cards you have before you pay for another.
Cold starts are the enemy. Pulling a 15 GB image and loading weights takes minutes. Scale-to-zero, the darling of serverless, quietly becomes scale-to-a-cold-start.
Utilisation is the KPI. Not request rate, not replica count. If the GPU sits at 20 % busy, the platform is wasting most of what it pays for.

What this series covers

This is the first of three posts written while building a GPU platform for a few dozen product teams. The next two get concrete:

Part 2 looks at how Kubernetes actually sees a GPU — device plugins, extended resources, and why nvidia.com/gpu: 1 is a much blunter instrument than it looks.
Part 3 is about sharing a GPU between workloads without regretting it: time-slicing, MIG and MPS, and how to choose.

If you only take one thing from this post: AI workloads are not special because of the model. They are special because the scarce resource moved, and the whole platform has to move with it.

A Kubernetes operator in a weekend

hello@example.com (Sam Ortega) — Sat, 14 Mar 2026 00:00:00 GMT

Every platform team has a script that someone runs by hand every Tuesday. Ours checked Prometheus during rollouts and pasted the graph into Slack. This is the story of turning it into a controller, and the three things the tutorials skipped.

Start from the resource, not the code

The temptation is to open the SDK scaffold and start typing. Resist it. The first hour should be spent on a YAML file that does not exist yet — the resource a user would write if the feature already worked:

apiVersion: sentinel.example.com/v1alpha1
kind: RolloutAnalysis
metadata:
  name: checkout
spec:
  target:
    kind: Deployment
    name: checkout
  metrics:
    - name: p99-latency
      query: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{app="checkout"}[5m]))
      maxValue: 0.25
  interval: 30s
  failureLimit: 3

If you cannot write this file, you do not know what you are building. Once you can, the controller almost writes itself: watch these, query that, compare, act.

The reconcile loop is the whole program

Everything an operator does lives in one function that gets called with a name and returns “done” or “try again later”. The scaffold hides that behind a lot of generated code; the part you own is this:

1
func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
2
  var analysis sentinelv1alpha1.RolloutAnalysis
3
  if err := r.Get(ctx, req.NamespacedName, &analysis); err != nil {
4
    return ctrl.Result{}, client.IgnoreNotFound(err)
5
  }
6

7
  verdict, err := r.evaluate(ctx, analysis.Spec.Metrics)
8
  if err != nil {
9
    return ctrl.Result{RequeueAfter: analysis.Spec.Interval.Duration}, err
10
  }
11

12
  analysis.Status.LastVerdict = verdict
13
  analysis.Status.Failures = countFailures(analysis.Status, verdict)
14
  if analysis.Status.Failures >= analysis.Spec.FailureLimit {
15
    if err := r.pauseRollout(ctx, analysis.Spec.Target); err != nil {
16
      return ctrl.Result{}, err
17
    }
18
    r.Recorder.Event(&analysis, "Warning", "RolloutPaused", "metric regression detected")
19
  }
20

21
  if err := r.Status().Update(ctx, &analysis); err != nil {
22
    return ctrl.Result{}, err
23
  }
24
  return ctrl.Result{RequeueAfter: analysis.Spec.Interval.Duration}, nil
25
}

The highlighted block is the only part that does anything. Everything else is fetching state and writing it back. That ratio is normal; if your reconciler is mostly action, it is probably not idempotent.

Three things the tutorials skipped

1. Status is an API, not a log

Early versions wrote a human-readable message into status. Users then wrote scripts that grep’d that message, and the first wording change broke them. Status needs the same discipline as Spec: typed fields, conditions with stable reasons, and a promise not to change them casually.

2. Requeue is not a retry

RequeueAfter says “look at this again in 30 s whether or not anything changed”. Returning an error says “something went wrong, back off exponentially”. Mixing them up gives you either a controller that hammers Prometheus during an outage or one that never notices the outage ended.

3. Test with envtest, then with a real cluster, then with a bad one

envtest spins up a real API server without nodes and catches every schema and RBAC mistake in seconds. What it does not catch is the behaviour when Prometheus returns a 502, when the target deployment is deleted mid-analysis, or when two analyses point at one deployment. Those need a kind cluster and deliberate sabotage:

kubectl -n monitoring scale deploy/prometheus --replicas=0
kubectl get rolloutanalysis checkout -w

Watching the status conditions flip to MetricsUnavailable and back — without pausing anything — was the moment the thing felt finished.

Was it a weekend?

The first working version, yes. The version that could be trusted with production rollouts took another three weekends and one incident. That version became Rollout Sentinel, and the Tuesday script has been deleted.

Talk: tail latency and hedged requests

hello@example.com (Sam Ortega) — Wed, 08 Oct 2025 00:00:00 GMT

Slides from a meetup talk on cutting p99 latency in a metrics store by sending a second request when the first one is slow.

What a year on call taught me

hello@example.com (Sam Ortega) — Mon, 20 Jan 2025 00:00:00 GMT

I joined the on-call rotation in January with a fresh laptop and a strong opinion that alerts should be fixed at the source. A year later the opinion survives, but it has company.

The first month I treated every page as a fact about the system. By March I understood that every page is a fact about a person’s decision, made months ago, about what they thought would matter. Half of ours were wrong, and nobody had been given permission to delete them.

So we gave permission. One rule: an alert that pages must have a runbook that says what to do, and if the answer is “look and see”, it is not a page, it is a dashboard.

Forty-one pages in a year, down from an estimated three hundred the year before. The three that mattered still paged.

Things I believed and no longer do

“Alert on symptoms, not causes.” Half true. Alert on symptoms to page; alert on causes to annotate, so the person who is paged sees the likely cause next to the symptom instead of hunting for it at 3 a.m.
“Automate the response.” Only after the third time. The first time you do not understand it. The second time you understand it but you are wrong. The third time, maybe.
“On-call should be a burden shared equally.” Shared, yes. Equally, no. The person who built the thing should carry more of it for the first quarter. Nothing improves a design like being woken up by it.

The three real incidents

I will not relive them here, but each one left a rule behind.

The certificate. Every expiring thing gets a calendar reminder thirty days out and an alert seven days out. Both. The alert alone was silenced during an unrelated incident and nobody unsilenced it.
The autoscaler. A scale-down policy that was correct for the traffic pattern of the month it was written. Now every scaling policy has a comment with the date and the assumption, so the next person knows what it was solving for.
The upgrade. We upgraded the control plane on a Friday because “it’s just a patch release”. It is never just a patch release. Upgrades go out Tuesday to Thursday, before noon, with someone watching who did not do the upgrade.

The system is not the thing in the cluster. The system is the cluster plus everyone who touches it, including the version of you that is asleep.

What I’d tell January me

Learn the runbooks before you need them. Fix one alert every week, not all of them at once. Write the incident review the same day, while it hurts. And keep a file of the pages that were right — on the bad weeks, that file is the reason you are still doing this.