Kubernetes Probes: Stop Accidental Restart Loops

A health check can make an outage worse if it restarts containers that are merely busy or temporarily unable to reach a dependency. Kubernetes startup, readiness and liveness probes answer different questions. Design the response to each failure before copying a probe configuration into a deployment.

The objective is to keep a recoverable application running, route traffic only when it can serve users and restart only when restarting is likely to help. A probe that returns a success code is not automatically a useful health signal.

Give each probe a distinct meaning

The Kubernetes probe guide describes three behaviors: startup probes protect slow initialization, readiness probes influence whether a Pod receives Service traffic, and liveness failures can trigger container restarts. Startup success allows the other probes to begin.

Translate those mechanisms into application questions. Has initialization finished? Can this instance serve the required request class? Is the process stuck in a condition that restarting can resolve?

Write these definitions beside the endpoint implementation. Otherwise, an endpoint named /health can gradually accumulate every dependency check and become unsuitable for liveness.

Keep liveness focused on recoverability

If a shared database becomes unavailable, restarting every application instance may not restore it. The restarts can increase load through reconnection attempts and repeated initialization. A dependency outage therefore needs careful treatment rather than automatic inclusion in liveness.

Choose a liveness signal that reflects an unrecoverable local condition, such as a service that can no longer make progress. Keep it cheap and independent of expensive downstream requests where possible.

Don't use a detailed diagnostic endpoint that performs many remote calls as a frequent probe. Diagnostics serve operators; probes drive automatic actions. Their operational budgets and failure meanings are different.

Use readiness to control traffic

A process can be alive but unable to handle its current workload. Readiness can represent that condition without forcing a restart. Define which dependencies are essential to serving requests and which allow degraded operation.

For example, losing an optional analytics destination might not make the service unready. Losing the only store needed for all requests might. Base the decision on the application's actual serving contract.

Beware of checks that remove every replica during a shared failure. Consider what the remaining load balancer response will be and how the service returns to readiness. The design should explain the intended user behavior during partial and total dependency outages.

Budget startup separately

Slow initialization is not the same as a hung application. If startup loads a large model, warms a cache or performs a controlled migration check, configure an appropriate startup budget from observed timings.

An illustrative fragment is:

startupProbe:
  httpGet:
    path: /startup
    port: 8080
  periodSeconds: 5
  failureThreshold: 24

These numbers illustrate a possible configuration, not a recommendation for every workload. Measure cold starts under realistic resource limits and set the failure policy accordingly.

Use YAML Validator to check a sanitized fragment's syntax. Validate the complete manifest with Kubernetes as well; syntax alone cannot establish that the resource configuration is accepted.

Choose thresholds from observations

Collect startup durations, ordinary probe response times and behavior during load. A timeout below the normal tail latency can mark healthy instances as failed. A very long timeout can delay detection of an actual local problem.

Document the intended detection window and the cost of false positives. Consider period, failure threshold, timeout and recovery threshold together. Don't adjust one value randomly after each incident.

Check the application server's capacity to respond to probes under load. If health requests share the same saturated worker pool as expensive user requests, the probe may measure queue pressure rather than a deadlock.

Test controlled failures in staging

Test slow startup, a temporarily unavailable dependency, a deliberate readiness failure and a controlled local hang. Observe traffic routing, restart counts and recovery behavior. Keep the experiments confined to an environment where the failure is authorized.

Capture relevant events and a short timeline. Use Text Diff to compare sanitized manifest revisions and JSON Viewer for structured test results. Retain the thresholds used in each experiment.

Don't accept a test solely because the container eventually became ready. Check whether it restarted unnecessarily and whether other replicas received enough traffic to become unhealthy themselves.

Keep probe endpoints narrow

Avoid returning secrets, database connection details or full stack traces from health endpoints. The response should reveal only what the relevant check needs. Detailed diagnostics can live behind a separate authorized operator interface.

Confirm that the probe targets the correct port, path and network listener. An endpoint accessible on localhost inside the application may not be accessible through the Pod address used by the probe.

Review configuration changes alongside application releases. A new startup task or a renamed route can invalidate old thresholds and paths. Keep the probe contract part of the service's release checklist.

Conclusion

Useful probes have different meanings: startup protects initialization, readiness controls serving and liveness addresses recoverable local failure. Measure timing, test controlled outages and inspect automatic actions. A health-check design succeeds when it improves recovery without creating a restart cascade.