A Kubernetes operator in a weekend

Every platform team has a script that someone runs by hand every Tuesday. Ours checked Prometheus during rollouts and pasted the graph into Slack. This is the story of turning it into a controller, and the three things the tutorials skipped.

Start from the resource, not the code

The temptation is to open the SDK scaffold and start typing. Resist it. The first hour should be spent on a YAML file that does not exist yet — the resource a user would write if the feature already worked:

apiVersion: sentinel.example.com/v1alpha1
kind: RolloutAnalysis
metadata:
  name: checkout
spec:
  target:
    kind: Deployment
    name: checkout
  metrics:
    - name: p99-latency
      query: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{app="checkout"}[5m]))
      maxValue: 0.25
  interval: 30s
  failureLimit: 3

If you cannot write this file, you do not know what you are building. Once you can, the controller almost writes itself: watch these, query that, compare, act.

The reconcile loop is the whole program

Everything an operator does lives in one function that gets called with a name and returns “done” or “try again later”. The scaffold hides that behind a lot of generated code; the part you own is this:

1
func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
2
  var analysis sentinelv1alpha1.RolloutAnalysis
3
  if err := r.Get(ctx, req.NamespacedName, &analysis); err != nil {
4
    return ctrl.Result{}, client.IgnoreNotFound(err)
5
  }
6

7
  verdict, err := r.evaluate(ctx, analysis.Spec.Metrics)
8
  if err != nil {
9
    return ctrl.Result{RequeueAfter: analysis.Spec.Interval.Duration}, err
10
  }
11

12
  analysis.Status.LastVerdict = verdict
13
  analysis.Status.Failures = countFailures(analysis.Status, verdict)
14
  if analysis.Status.Failures >= analysis.Spec.FailureLimit {
15
    if err := r.pauseRollout(ctx, analysis.Spec.Target); err != nil {
16
      return ctrl.Result{}, err
17
    }
18
    r.Recorder.Event(&analysis, "Warning", "RolloutPaused", "metric regression detected")
19
  }
20

21
  if err := r.Status().Update(ctx, &analysis); err != nil {
22
    return ctrl.Result{}, err
23
  }
24
  return ctrl.Result{RequeueAfter: analysis.Spec.Interval.Duration}, nil
25
}

The highlighted block is the only part that does anything. Everything else is fetching state and writing it back. That ratio is normal; if your reconciler is mostly action, it is probably not idempotent.

Three things the tutorials skipped

1. Status is an API, not a log

Early versions wrote a human-readable message into status. Users then wrote scripts that grep’d that message, and the first wording change broke them. Status needs the same discipline as Spec: typed fields, conditions with stable reasons, and a promise not to change them casually.

2. Requeue is not a retry

RequeueAfter says “look at this again in 30 s whether or not anything changed”. Returning an error says “something went wrong, back off exponentially”. Mixing them up gives you either a controller that hammers Prometheus during an outage or one that never notices the outage ended.

3. Test with envtest, then with a real cluster, then with a bad one

envtest spins up a real API server without nodes and catches every schema and RBAC mistake in seconds. What it does not catch is the behaviour when Prometheus returns a 502, when the target deployment is deleted mid-analysis, or when two analyses point at one deployment. Those need a kind cluster and deliberate sabotage:

kubectl -n monitoring scale deploy/prometheus --replicas=0
kubectl get rolloutanalysis checkout -w

Watching the status conditions flip to MetricsUnavailable and back — without pausing anything — was the moment the thing felt finished.

Was it a weekend?

The first working version, yes. The version that could be trusted with production rollouts took another three weekends and one incident. That version became Rollout Sentinel, and the Tuesday script has been deleted.