$Kubernetes Probes in Detail: Liveness, Readiness, and Startup

I deployed Stirling-PDF to my homelab cluster this week, and the first thing it greeted me with was a 502 Bad Gateway. The pod was Running. The Service had endpoints. The Ingress was fine. TLS was fine. And yet 502.

The culprit was a missing probe, and the fix is a good excuse to write down how Kubernetes health probes actually work, because the three of them - liveness, readiness, and startup answer three different questions and trigger three different consequences. Mixing them up is one of the most common ways to make a cluster less reliable while trying to make it more reliable.

The incident, in one paragraph

Stirling-PDF is a Spring Boot application. It takes about 60–90 seconds from container start to actually listening on its port. Kubernetes, by default, considers a container "ready" the moment its process starts. So the pod IP was registered in the Service's endpoints immediately, Traefik dutifully forwarded my request to port 8080, nothing was listening there yet, and Traefik returned 502. Ninety seconds later the app finished booting and everything "fixed itself." That 502 window would reopen on every restart, every node drain, every image update forever unless the cluster learns to tell "process started" apart from "application ready." That's exactly what probes are for.

What a probe is

A probe is a periodic check the kubelet (the node agent, not the API server) runs against your container. Each check either passes or fails. What happens on failure depends on which of the three probe types it is.

There are four check mechanisms, usable with any probe type:

• httpGet - HTTP request to a path/port. Any status in 200–399 passes. The most common choice for web apps.
• tcpSocket - passes if a TCP connection can be opened. Useful for non-HTTP services (databases, SMTP), but weak evidence: a process can accept connections and still be wedged.
• exec - runs a command inside the container; exit code 0 passes. Most flexible, most expensive (a process fork per check, per container, forever).
• grpc - native gRPC health-check protocol, for services that implement grpc.health.v1.Health.

And five tuning knobs, per probe:

• periodSeconds - how often to check (default 10).
• timeoutSeconds - how long to wait for an answer (default 1 second; genuinely too tight for many JVM apps under load; this default has caused countless mystery restarts).
• failureThreshold - consecutive failures before the probe is considered failed (default 3).
• successThreshold - consecutive passes needed to recover (must be 1 for liveness and startup; can be higher for readiness).
• initialDelaySeconds - wait before the first check (default 0). Mostly obsolete now that startup probes exist; more on that below.

Liveness: "is the process alive, or wedged?"

On failure: the kubelet kills and restarts the container (subject to the pod's restartPolicy).

Liveness answers exactly one question: is this process so broken that a restart is the cure? Deadlocked threads, a hung JVM, an event loop stuck in an infinite loop — these are liveness failures. The process exists, so Kubernetes can't see anything wrong from the outside; the probe is how the app tells the kubelet "I'm beyond saving, reboot me."

The most important rule of liveness probes: the endpoint must be cheap and must not check dependencies. If your liveness endpoint pings the database, then a database outage makes every replica of your app fail liveness simultaneously, and the kubelet restarts all of them, in a loop, for as long as the DB is down. You've converted "database down, app serving degraded responses" into "database down AND app in a permanent restart storm." The restart fixed nothing because the problem was never in the process.

The second most important rule: be generous. A liveness probe that's too aggressive creates a death spiral. Picture an app under heavy load: responses slow down, the probe (with its 1-second default timeout) starts timing out, kubelet restarts the container, the restart throws away every warm cache and JIT compilation, the cold container is even slower, the probe fails again. Aggressive liveness turns "slow" into "down." Long period, real timeout (2–5s), failureThreshold of at least 3.

Some people run no liveness probe at all, and that's a defensible position: if your app doesn't have a known wedge-mode, a liveness probe is pure downside risk. Add it when you have a failure mode a restart actually cures.

Readiness: "can this pod take traffic right now?"

On failure: the pod's IP is removed from the Service's endpoints. No restart. Nothing is killed. Traffic just stops arriving until the probe passes again.

This is the probe people misunderstand most, because unlike liveness it isn't about health; it's about traffic routing. A pod failing readiness is a pod saying "not right now": still booting, warming a cache, re-establishing a connection, momentarily overloaded. The condition is expected to clear on its own, and when it does, the endpoint is re-added and traffic resumes. Restarting would be counterproductive; which is precisely why readiness doesn't restart anything.

Readiness is also what makes zero-downtime rolling deploys work. During a rollout, the new pod receives traffic only after its readiness probe passes, and the old pod is only terminated once the new one is ready. Without a readiness probe, every deploy shifts traffic onto cold pods and produces a blip of errors. I watched this work correctly right after adding probes to Stirling-PDF: kubectl rollout status sat on "1 old replicas are pending termination" for a minute; that was the old pod holding traffic while the new one booted behind the scenes. The 502 window was gone.

Should readiness check dependencies? This one is legitimately debated. The case for: if the pod can't reach the database, it genuinely can't serve, so why route traffic to it? The case against: if the shared database dies, every pod fails readiness at once, the Service empties, and clients get connection refused instead of a meaningful 503 from your app — you've made the outage's blast radius wider and its diagnosis harder. My rule: readiness may check things that are per-pod (my connection pool, my local cache warmed), but shared-fate dependencies belong in application-level error handling, not in the probe.

Startup: "is it still booting?"

While the startup probe hasn't yet passed, liveness and readiness probes are disabled. Once it passes (once), it never runs again for that container's lifetime. If it exhausts its failureThreshold, the container is killed and restarted.

Startup probes exist to resolve a conflict the other two can't: slow boot versus tight liveness. My Stirling-PDF takes up to 90 seconds to boot. A liveness probe with default settings would kill it at ~30 seconds, forever, in a loop; the pod would never come up at all. The old workaround was initialDelaySeconds: 120 on the liveness probe, but that's a bad trade: it delays hang detection by two minutes for the whole life of the container, boots included, even though boot only needs the grace once. And if boot occasionally takes 130 seconds, you're dead anyway.

The startup probe fixes this properly:

startupProbe:
  httpGet:
    path: /api/v1/info/status
    port: 8080
  periodSeconds: 5
  failureThreshold: 36

5 × 36 = 180 seconds of boot budget, checked every 5 seconds. Boot finishes in 60 seconds? The startup probe passes at 60 seconds and liveness/readiness take over immediately; the budget is a ceiling, not a wait. Boot hangs past 3 minutes? Kill and retry, which is the right call for a boot that's genuinely stuck.

The rule of thumb: any app whose boot time is longer than liveness periodSeconds × failureThreshold needs a startup probe. For JVM apps, that's nearly always.

The full picture for Stirling-PDF

startupProbe:            # boot budget: up to 180s, polls every 5s
  httpGet:
    path: /api/v1/info/status
    port: 8080
  periodSeconds: 5
  failureThreshold: 36
readinessProbe:          # gates Service endpoints
  httpGet:
    path: /api/v1/info/status
    port: 8080
  periodSeconds: 10
livenessProbe:           # restarts a hung JVM after 90s of silence
  httpGet:
    path: /api/v1/info/status
    port: 8080
  periodSeconds: 30
  failureThreshold: 3

All three probes hit the same endpoint here, and that's fine because the endpoint is a dumb "am I serving HTTP" check with no dependencies. The moment your readiness check wants to verify something your liveness check must not (like a DB connection), split the endpoints ( Spring Boot's /actuator/health/liveness and /actuator/health/readiness ) exist for exactly this reason.

Cheat sheet

Three probes, three questions, three consequences. Get the mapping right and Kubernetes quietly absorbs restarts, deploys, and hangs. Get it wrong and it will happily amplify your outages instead.