Skip to content
SP.
EN DE
Back to Notes
Runbooks

When a rollout stops progressing

  • Kubernetes
  • incident-response
  • delivery

Signal

A deployment exceeds its progress deadline, or new pods remain unready while the previous replica set continues serving traffic. This is a general-purpose runbook; adapt names, namespaces, and escalation paths to your environment.

Establish scope

Confirm the cluster context and namespace before inspecting the rollout. These commands read state; they do not change the deployment.

kubectl config current-context
kubectl -n <namespace> rollout status deployment/<app> --timeout=30s
kubectl -n <namespace> describe deployment <app>
kubectl -n <namespace> get pods -l app=<app> -o wide

Diagnose

  1. Check pod events for image-pull errors, unschedulable resources, or failed mounts.
  2. Inspect readiness failures and application logs. Do not expose secrets when sharing logs.
  3. Compare the deployed image digest and configuration with the last healthy release.
  4. Check error rate and latency for existing users. A stalled rollout is not automatically an outage.

Recover

For a GitOps-managed deployment, revert the desired image digest in Git. An imperative rollback may be immediately undone by reconciliation. Follow the team’s emergency-change process if Git or the controller is unavailable.

Verify and close

Confirm all intended replicas are ready, the controller reports a synced state, and user-facing signals have returned to their normal range. Record the change, the failure mode, and the missing check that would have caught it earlier.

Back to Notes