WSS
On this page
Containers & OrchestrationRequiredUpdated

Pod lifecycle and probes

Managing container health and traffic routing using liveness, readiness, and startup probes.

What it is

In Kubernetes, a Pod goes through a distinct lifecycle: Pending, Running, Succeeded, Failed, or Unknown. Because a Pod is “Running” the moment its primary process starts, Kubernetes cannot inherently know if the application inside the container is actually ready to serve traffic or if it has entered a deadlocked state.

Probes are automated diagnostic checks performed periodically by the kubelet on a container to determine its actual health. There are three specific types of probes:

  • Liveness Probes: Determine if the container is deadlocked and needs to be restarted.
  • Readiness Probes: Determine if the container is ready to accept network traffic.
  • Startup Probes: Determine if a slow-starting application has finished its initial initialization.

Why it matters

Without probes, an orchestrator routes traffic blindly. If a Java application takes forty seconds to warm up its cache, a naive orchestrator sends user traffic to it immediately after boot, resulting in forty seconds of 502 Bad Gateway errors.

If an application deadlocks due to a thread exhaustion bug, its main process does not exit. The operating system considers the container perfectly healthy. Without a liveness probe, that deadlocked container remains in rotation forever, dropping all requests sent to it until a human intervenes. Probes provide the telemetry required for true self-healing architectures.

How to implement

Probes can execute commands inside the container, open a TCP socket, or make an HTTP request. HTTP requests are the most common and robust method.

Liveness Probe

Define a liveness probe to check an endpoint that strictly evaluates the internal state of the application. If this probe fails, the kubelet kills the container and restarts it according to its restart policy.

livenessProbe:
  httpGet:
    path: /health/liveness
    port: 8080
  initialDelaySeconds: 3
  periodSeconds: 10

Readiness Probe

Define a readiness probe to check if the application can reach its critical dependencies (like a database). If a readiness probe fails, the container is NOT killed. Instead, the endpoint controller removes the Pod’s IP address from all Services matching the Pod. The Pod stops receiving external traffic until the readiness probe passes again.

readinessProbe:
  httpGet:
    path: /health/readiness
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

Startup Probe

For legacy applications that take unpredictably long times to start (e.g., three minutes to run database migrations), a startup probe protects them. While a startup probe is running, liveness and readiness probes are disabled. Once the startup probe succeeds once, it never runs again, handing control over to the liveness and readiness probes.

Common mistakes

Deep Liveness Probes

A liveness probe should never check external dependencies. If your liveness probe queries the database, and the database has a momentary hiccup, the liveness probe fails. The kubelet will aggressively restart every single web server in the cluster simultaneously, causing a massive, self-inflicted cascading failure. External dependencies should only be checked by readiness probes.

Missing Initial Delays

If a Spring Boot application takes 30 seconds to start, but the liveness probe has no initialDelaySeconds, the kubelet will begin checking the endpoint immediately. The application will fail the check, and the kubelet will kill it before it finishes booting. The container enters an infinite CrashLoopBackOff state and never successfully starts.

Identical Probes

Using the exact same /health endpoint for both liveness and readiness defeats their distinct purposes. If the database goes down, you want the readiness probe to fail (removing the Pod from the load balancer) but the liveness probe to pass (keeping the process running so it can reconnect when the database returns).

Verification

Deploy a Pod with probes configured. Use kubectl describe pod <pod-name> to monitor the lifecycle events.

During boot, the Events section will show the kubelet executing the probes. If a probe fails, you will see explicit warning events:

Warning  Unhealthy  2m   kubelet  Readiness probe failed: HTTP probe failed with statuscode: 503
Warning  Unhealthy  2m   kubelet  Liveness probe failed: Get "http://10.0.1.5:8080/health": dial tcp 10.0.1.5:8080: connect: connection refused

Verify that traffic routing reacts to the readiness probe by watching the Endpoints object:

kubectl get endpoints <service-name> --watch

When the readiness probe fails, the Pod’s IP address will be dynamically removed from the Endpoints list. When it passes, it will be added back.

Related topics

Sources & further reading