Liveness and readiness probes in Microservices - System Design Guide

Problem Statement

Without proper health checks, a microservice might appear healthy while it is stuck or unable to serve traffic, causing requests to fail or time out. Also, routing traffic to a service that is still starting or temporarily unable to handle requests leads to poor user experience and wasted resources.

Solution

Liveness probes regularly check if a service is alive and responsive; if not, the system restarts it to recover. Readiness probes check if a service is ready to accept traffic, preventing routing requests to it until it is fully prepared. Together, they ensure only healthy and ready services receive traffic, improving reliability and availability.

Architecture

Load

Balancer

→Readiness

↓

Liveness Probe

Liveness Probe

This diagram shows the load balancer sending traffic only to service instances that pass the readiness probe. The liveness probe monitors the service health and triggers restarts if needed.

Trade-offs

✓ Pros

→

Automatically recovers services stuck in unhealthy states by restarting them.

→

Prevents routing traffic to services that are not ready, improving user experience.

→

Enables faster detection of failures and reduces downtime.

→

Improves overall system reliability and availability.

✗ Cons

→

Requires careful configuration to avoid false positives causing unnecessary restarts.

→

Adds complexity to deployment and monitoring setup.

→

Improper probe design can mask real issues or delay recovery.

Use when deploying microservices in orchestrated environments like Kubernetes with frequent deployments and dynamic scaling, especially when services have startup delays or can get stuck.

Avoid if your service is extremely simple, stateless, and fast to start, or if you have no orchestration platform to act on probe results.

Real World Examples

Google

Kubernetes uses liveness and readiness probes to manage container lifecycle, ensuring only healthy pods receive traffic and restarting unhealthy ones automatically.

Netflix

Netflix uses readiness probes to prevent routing user requests to instances still warming up or temporarily overloaded, improving streaming reliability.

Uber

Uber employs liveness probes to detect and restart microservices stuck due to deadlocks or resource exhaustion, maintaining high availability.

Code Example

The before code has no health checks, so the orchestrator cannot detect if the service is stuck or not ready. The after code adds two endpoints: /health/liveness to confirm the service is alive, and /health/readiness to indicate if it is ready to serve traffic. The readiness endpoint returns 503 until the service finishes initialization, preventing traffic routing prematurely.

Microservices

### Before: No probes, service always assumed healthy
from flask import Flask
app = Flask(__name__)

@app.route('/')
def home():
    return 'Hello World'

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)


### After: Adding liveness and readiness endpoints
from flask import Flask, jsonify
app = Flask(__name__)

service_ready = False

@app.route('/')
def home():
    if not service_ready:
        return 'Service not ready', 503
    return 'Hello World'

@app.route('/health/liveness')
def liveness():
    # Check if service process is alive
    return jsonify(status='alive')

@app.route('/health/readiness')
def readiness():
    # Check if service is ready to serve traffic
    if service_ready:
        return jsonify(status='ready')
    else:
        return jsonify(status='not ready'), 503

# Simulate readiness after some initialization
import threading, time
def set_ready():
    global service_ready
    time.sleep(5)  # simulate startup delay
    service_ready = True
threading.Thread(target=set_ready).start()

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

Alternatives

External Health Monitoring

Uses an external system to check service health rather than internal probes.

Use when: When you want centralized monitoring across many services and infrastructure, or when probes cannot be embedded in the service.

Circuit Breaker

Prevents calls to failing services based on error rates rather than probing service health directly.

Use when: When you want to protect clients from cascading failures rather than managing service lifecycle.

Summary

Liveness and readiness probes prevent routing traffic to unhealthy or unready services and enable automatic recovery.

They improve system reliability by detecting failures early and avoiding downtime.

Proper configuration and understanding of their differences are essential for effective use.

Practice

1. What is the main purpose of a liveness probe in microservices?

easy

A. To check if the service is ready to accept traffic

B. To log user requests for debugging

C. To monitor the network latency between services

D. To check if the service is alive and restart it if it is not

Solution

1. Step 1: Understand the role of liveness probes

   Liveness probes detect if a service is stuck or dead and need restarting.

2. Step 2: Differentiate from readiness probes

   Readiness probes check if the service can handle requests, not if it is alive.

3. Final Answer:

   To check if the service is alive and restart it if it is not -> Option D

4. Quick Check:

   Liveness probe = check alive and restart [OK]

Hint: Liveness = alive and restart, Readiness = ready for traffic [OK]

Common Mistakes:

• Confusing liveness with readiness probes
• Thinking liveness probes check traffic readiness
• Assuming liveness probes monitor performance

2. Which of the following is the correct syntax to define a readiness probe in a Kubernetes pod spec?

easy

A. livenessProbe:
exec:
command: ["cat", "/tmp/healthy"]
timeoutSeconds: 1

B. livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 5
periodSeconds: 10

C. readinessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 10

D. livenessProbe:
httpGet:
path: /ready
port: 8080
failureThreshold: 3

Solution

1. Step 1: Identify readiness probe syntax

   Readiness probes often use httpGet with path and port, plus delay and period settings.

2. Step 2: Confirm correct fields and indentation

   readinessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 5 periodSeconds: 10 correctly shows readinessProbe with httpGet, initialDelaySeconds, and periodSeconds.

3. Final Answer:

   readinessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 5 periodSeconds: 10 -> Option C

4. Quick Check:

   Readiness probe syntax = readinessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 5 periodSeconds: 10 [OK]

Hint: Readiness uses httpGet with path and port in YAML [OK]

Common Mistakes:

• Mixing livenessProbe and readinessProbe fields
• Incorrect indentation in YAML
• Using wrong probe type for readiness

3. Given this Kubernetes pod spec snippet, what will happen if the readiness probe fails continuously?

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 3

medium

A. The pod will be restarted immediately

B. The pod will be marked as not ready and removed from service endpoints

C. The pod will ignore the failure and continue serving traffic

D. The pod will scale up automatically

Solution

1. Step 1: Understand readiness probe failure effect

   Readiness probe failure marks pod as not ready, so it stops receiving traffic.

2. Step 2: Differentiate from liveness probe effect

   Liveness probe failure triggers pod restart, readiness does not.

3. Final Answer:

   The pod will be marked as not ready and removed from service endpoints -> Option B

4. Quick Check:

   Readiness failure = pod not ready, no restart [OK]

Hint: Readiness failure removes pod from load balancer, no restart [OK]

Common Mistakes:

• Confusing readiness failure with pod restart
• Assuming pod scales automatically on probe failure
• Ignoring failureThreshold effect

4. A microservice has a liveness probe configured as an HTTP GET on /health. The service sometimes returns HTTP 500 during startup but is healthy afterward. What is the best fix to avoid unnecessary restarts?

medium

A. Increase initialDelaySeconds to allow startup time before probing

B. Change the probe to readiness probe instead of liveness probe

C. Remove the probe completely to avoid restarts

D. Set failureThreshold to 1 to detect failures faster

Solution

1. Step 1: Identify cause of restarts

   Liveness probe fails during startup because service returns HTTP 500 before ready.

2. Step 2: Adjust probe timing to avoid false failures

   Increasing initialDelaySeconds delays probe start, allowing service to become healthy first.

3. Final Answer:

   Increase initialDelaySeconds to allow startup time before probing -> Option A

4. Quick Check:

   Delay liveness probe start to avoid false failures [OK]

Hint: Delay liveness probe start to avoid false failure during startup [OK]

Common Mistakes:

• Removing probes which reduces reliability
• Confusing readiness and liveness probe roles
• Setting failureThreshold too low causing quick restarts

5. You have a microservice that takes time to initialize resources before it can serve requests. You want to ensure it is not restarted unnecessarily but also not receive traffic before ready. How should you configure liveness and readiness probes?

hard

A. Set liveness probe with a longer initialDelaySeconds and readiness probe to check resource initialization

B. Use only a liveness probe with a short periodSeconds to restart fast

C. Use only a readiness probe and no liveness probe

D. Set both probes to the same HTTP path and timing

Solution

1. Step 1: Prevent unnecessary restarts during initialization

   Set liveness probe initialDelaySeconds long enough to avoid restarting while initializing.

2. Step 2: Use readiness probe to block traffic until ready

   Readiness probe should check if resources are initialized before accepting traffic.

3. Final Answer:

   Set liveness probe with a longer initialDelaySeconds and readiness probe to check resource initialization -> Option A

4. Quick Check:

   Liveness delay + readiness check = safe startup [OK]

Hint: Delay liveness, readiness blocks traffic until ready [OK]

Common Mistakes:

• Using only one probe type causing traffic or restart issues
• Setting same path and timing for both probes
• Not delaying liveness probe causing premature restarts