Skip to main content

Kubernetes Deployment Monitoring for Incident Investigation

A Kubernetes deployment timeline should answer more than “the rollout completed.” During an incident, engineers need to know exactly which revision ran on which pods and nodes, how that state changed, and whether the failure follows the workload, the node, the namespace, or a dependency.

Capture rollout evidence at deployment time​

At minimum, retain the workload name, namespace, cluster, image digest, deployment revision, commit or artifact reference, manifest or Git revision, rollout start and completion times, and the actor or automation that initiated it. Include configuration and secret version references where exposing values would be unsafe.

This lets the team distinguish a new image from a config-only rollout, a controller reconciliation, or an unrelated node event.

The first five checks after a regression​

These read-only commands establish a useful initial picture:

kubectl rollout status deployment/checkout-api -n production
kubectl rollout history deployment/checkout-api -n production
kubectl get pods -n production -l app=checkout-api -o wide
kubectl get events -n production --sort-by=.lastTimestamp
kubectl describe deployment/checkout-api -n production

Compare the output with a healthy replica or region. Look for revision skew, restarts, scheduling changes, image-pull errors, probe failures, OOMKilled, node pressure, PVC events, and unexpected traffic or replica changes.

Separate workload evidence from node evidence​

A failure that follows a deployment revision across nodes suggests a workload or configuration candidate. A failure isolated to one node, availability zone, volume, or CNI path points somewhere else. This distinction is more useful than the chronological fact that a deployment happened nearby.

For node-level candidates, capture the node condition and affected pod placement before rescheduling or draining. Remediation may be appropriate, but losing the before-state makes later diagnosis weaker.

Add service, environment, namespace, workload, pod, node, and revision labels consistently to logs, metrics, traces, and deploy markers. Avoid high-cardinality labels such as raw request IDs in metrics. The goal is to move from a user symptom to a workload revision and then to the CI artifact without manually translating names between tools.

Where OctoLaunch fits​

OctoLaunch is intended to connect Kubernetes events and workload state with CI/CD, source-control, and runtime metadata. Source data remains in the systems that produced it. Treat ranked explanations as hypotheses to inspect, not automatic production actions.

Continue with how to correlate deployments and incidents and incident correlation.