Skip to main content

How to Correlate Deployments and Incidents

The most common mistake in a production incident is blaming the latest deploy because it is visible and recent. A release is a strong lead, not a verdict. Use this workflow to decide whether it deserves rollback priority or whether the incident is better explained by infrastructure, dependency, or configuration evidence.

1. Freeze the timeline​

Record the incident start, first user impact, alert time, mitigation time, and recovery time in UTC. Then collect every change in a fixed window before the first impact. Start with 30–60 minutes and widen only when the system’s propagation delay justifies it.

Include more than application deployments:

  • application artifact, commit range, feature flags, and configuration revisions;
  • Kubernetes rollout, autoscaler, node, secret, and ingress changes;
  • database migrations, queue configuration, and dependency maintenance;
  • CI retries, failed promotion gates, and manual interventions.

If an event has no timestamp or immutable identifier, write that down. Missing provenance is an operational defect worth fixing after the incident.

2. Compare affected and unaffected populations​

Do not ask only “what changed?” Ask “what is different about the requests that fail?” Compare the affected service, route, region, node, tenant, release ring, and dependency path with a healthy control group.

For a Kubernetes workload, this could start with:

kubectl get pods -n production -l app=checkout-api -o wide
kubectl rollout history deployment/checkout-api -n production
kubectl get events -n production --sort-by=.lastTimestamp

The commands are evidence collection, not a remediation. Do not run destructive operations from a generic guide during an active incident.

3. Grade each candidate against four tests​

Give every leading candidate the same test:

  1. Precedence: did it occur before the symptom, with enough time for its effect to appear?
  2. Scope: does its blast radius match the failing population?
  3. Mechanism: can it plausibly create the observed error, latency, saturation, or availability pattern?
  4. Falsifiability: what observation would make the candidate less likely?

A release that precedes an incident but appears equally on healthy and unhealthy workloads has weak scope evidence. A node event that precedes the incident and is shared by every failing pod has stronger scope evidence, even if a release is more recent.

4. Preserve the raw evidence​

Save links or identifiers for the deployment record, commit, CI run, dashboard query, trace sample, log query, Kubernetes event, and incident ticket. Screenshots without a query or timestamp are hard to verify later.

At the end of the incident, attach a short conclusion that separates:

  • confirmed cause;
  • contributing conditions;
  • mitigations that restored service;
  • remaining uncertainty;
  • follow-up instrumentation work.

This distinction prevents a fast rollback from being written up as proof of a root cause it did not actually establish.

5. Improve the next investigation​

Add a release marker to telemetry, ensure artifacts carry commit and environment metadata, and define service ownership. These small pieces of plumbing reduce time-to-understanding far more reliably than adding another dashboard.

Use incident correlation for the evidence model, CI/CD monitoring for delivery data, and Kubernetes deployment monitoring for cluster-specific evidence.