Kubernetes restart loops: fix startup probes

Read the previous attempt, not only the current pod status

Record the namespace and exact pod name. Inspect events, container termination details, and the previous container’s logs. A current log stream may contain only the latest incomplete startup and miss the attempt that explains the restart.

Look for evidence such as repeated liveness-probe failures followed by a kill, a nonzero application exit with a stack trace, or an OOMKilled termination reason. These are different investigations. Increasing a probe delay cannot repair an invalid configuration file or a memory limit that is too small.

bash · EXAMPLE

kubectl -n example get pod example-api-abc123 -o wide
kubectl -n example describe pod example-api-abc123
kubectl -n example logs example-api-abc123 -c api --previous

Understand which probe is allowed to do what

A readiness failure removes a pod from ready service endpoints; it does not, by itself, restart that container. A liveness failure can trigger a restart. A startup probe protects an initialization period by holding back readiness and liveness probing until startup succeeds.

Choose the question each endpoint answers. “Can this instance receive traffic?” and “Is this process irrecoverably stuck?” are not interchangeable. A dependency outage should not automatically turn a liveness check into a machine for restarting every replica.

Measure startup time before choosing a budget

Compare timestamps from several cold starts in a test environment. Include the slow but legitimate cases: loading a large model, warming a cache, or initializing local state. Separate that measured work from waiting forever for an unavailable external service.

Decide what is acceptable for this workload and how a genuine startup failure should become visible. The example below allows roughly three minutes of failed startup probes, but it is a demonstration value, not a universal recommendation. Probe execution and scheduling also affect observed timing.

Use a startup probe instead of a permanently weak liveness check

This fragment belongs under one container in a pod template. The application must implement the paths with the intended meanings and listen on the named port. Do not paste probe paths that always return success just to make the pod green.

Once startup succeeds, the normal readiness and liveness checks apply. Keep liveness focused on failures a restart can reasonably repair. Apply changes through your usual manifest review and rollout process rather than editing a live pod that a controller will replace.

yaml · EXAMPLE

ports:
  - name: http
    containerPort: 8080
startupProbe:
  httpGet:
    path: /health/startup
    port: http
  periodSeconds: 5
  failureThreshold: 36
readinessProbe:
  httpGet:
    path: /health/ready
    port: http
  periodSeconds: 5
livenessProbe:
  httpGet:
    path: /health/live
    port: http
  periodSeconds: 10
  failureThreshold: 3

Verify behavior during startup and during a real fault

During a controlled cold start, confirm that the restart count stays stable while initialization proceeds and that readiness becomes true only when the instance can accept work. Then send a request through the service, not only directly to the pod.

In a disposable environment, separately test an application that fails to initialize and an instance that becomes unhealthy after startup. The first must eventually fail its startup budget; the second must follow the intended liveness behavior. A probe configuration is useful only if it distinguishes these cases.

Know the limits

Probe paths and timing values are illustrative. Multi-container pods, service meshes, overloaded nodes, resource pressure, and unavailable dependencies can introduce other causes. This guide does not treat CrashLoopBackOff as a diagnosis.

Sources & further reading

A technical publication by CoreTrailKit.

Browse all articles