Rollout Sentinel

The problem

Progressive delivery tools assume a service mesh. Most of our clusters did not have one, and the teams that did were not keen to route every request through it just to get canary analysis. What everyone did have was Prometheus.

What it does

Rollout Sentinel is a small controller with one custom resource, RolloutAnalysis. You point it at a Deployment, list the PromQL queries that define “healthy”, and set a failure budget. During a rollout it evaluates the queries every interval and, when the budget is spent, pauses the Deployment and emits an event.

apiVersion: sentinel.example.com/v1alpha1
kind: RolloutAnalysis
metadata:
  name: checkout
spec:
  target: { kind: Deployment, name: checkout }
  metrics:
    - name: p99-latency
      query: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{app="checkout"}[5m]))
      maxValue: 0.25
    - name: error-rate
      query: sum(rate(http_requests_total{app="checkout",code=~"5.."}[5m])) / sum(rate(http_requests_total{app="checkout"}[5m]))
      maxValue: 0.01
  interval: 30s
  failureLimit: 3

Design notes

• Pause, don’t roll back. A paused rollout keeps both versions running so an engineer can look at the graphs. Rolling back automatically hid the evidence.
• Status is typed. Conditions with stable reasons (MetricsUnavailable, RegressionDetected) so dashboards and scripts can rely on them.
• No admission webhook. The operator only ever writes to resources it was pointed at, and only to pause them. Nothing it does can block a deploy that a human wants.

Outcome

Running in front of about forty Deployments. Eleven paused rollouts in the first quarter, nine of them regressions that would otherwise have reached all users. The build log is in A Kubernetes operator in a weekend.