Liveness and readiness probes in Microservices - Architecture Diagram

System Overview - Liveness and readiness probes

This system uses liveness and readiness probes to monitor microservices health. Liveness probes check if a service is alive and should be restarted if not. Readiness probes check if a service is ready to accept traffic, ensuring smooth user experience.

Architecture Diagram

User
  |
  v
Load Balancer
  |
  v
API Gateway
  |
  v
+------------------+      +------------------+
| Microservice A    |<---->| Liveness Probe   |
| (with Readiness)  |      +------------------+
+------------------+
        |
        v
+------------------+
| Database         |
+------------------+

Components

User

client

Sends requests to the system

Load Balancer

load_balancer

Distributes incoming requests to API Gateway instances

API Gateway

api_gateway

Routes requests to microservices and performs health checks

Microservice A

service

Handles business logic; has liveness and readiness probes

Liveness Probe

health_check

Checks if Microservice A is alive; triggers restart if failed

Readiness Probe

health_check

Checks if Microservice A is ready to serve traffic

Database

database

Stores persistent data used by Microservice A

Request Flow - 9 Hops

User → Load Balancer

Load Balancer → API Gateway

API Gateway → Readiness Probe

API Gateway → Microservice A

Microservice A → Database

Microservice A → API Gateway

API Gateway → Load Balancer

Load Balancer → User

Liveness Probe → Microservice A

Failure Scenario

Component Fails:Microservice A

Impact:If Microservice A crashes, liveness probe detects failure and triggers restart. Readiness probe marks service as not ready, so no traffic is sent to it.

Mitigation:Orchestrator restarts Microservice A automatically. Load balancer and API Gateway stop sending traffic until readiness probe reports ready.

Architecture Quiz - 3 Questions

Which probe ensures the service is ready to accept user traffic?

A Liveness probe

B Readiness probe

C Load balancer health check

D Database connectivity check

Design Principle

Using liveness and readiness probes separates concerns of service health and traffic readiness. This improves system reliability by restarting unhealthy services and preventing traffic to services not ready to serve, ensuring smooth user experience.

Practice

1. What is the main purpose of a liveness probe in microservices?

easy

A. To check if the service is ready to accept traffic

B. To log user requests for debugging

C. To monitor the network latency between services

D. To check if the service is alive and restart it if it is not

Solution

1. Step 1: Understand the role of liveness probes

   Liveness probes detect if a service is stuck or dead and need restarting.

2. Step 2: Differentiate from readiness probes

   Readiness probes check if the service can handle requests, not if it is alive.

3. Final Answer:

   To check if the service is alive and restart it if it is not -> Option D

4. Quick Check:

   Liveness probe = check alive and restart [OK]

Hint: Liveness = alive and restart, Readiness = ready for traffic [OK]

Common Mistakes:

• Confusing liveness with readiness probes
• Thinking liveness probes check traffic readiness
• Assuming liveness probes monitor performance

2. Which of the following is the correct syntax to define a readiness probe in a Kubernetes pod spec?

easy

A. livenessProbe:
exec:
command: ["cat", "/tmp/healthy"]
timeoutSeconds: 1

B. livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 5
periodSeconds: 10

C. readinessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 10

D. livenessProbe:
httpGet:
path: /ready
port: 8080
failureThreshold: 3

Solution

1. Step 1: Identify readiness probe syntax

   Readiness probes often use httpGet with path and port, plus delay and period settings.

2. Step 2: Confirm correct fields and indentation

   readinessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 5 periodSeconds: 10 correctly shows readinessProbe with httpGet, initialDelaySeconds, and periodSeconds.

3. Final Answer:

   readinessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 5 periodSeconds: 10 -> Option C

4. Quick Check:

   Readiness probe syntax = readinessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 5 periodSeconds: 10 [OK]

Hint: Readiness uses httpGet with path and port in YAML [OK]

Common Mistakes:

• Mixing livenessProbe and readinessProbe fields
• Incorrect indentation in YAML
• Using wrong probe type for readiness

3. Given this Kubernetes pod spec snippet, what will happen if the readiness probe fails continuously?

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 3

medium

A. The pod will be restarted immediately

B. The pod will be marked as not ready and removed from service endpoints

C. The pod will ignore the failure and continue serving traffic

D. The pod will scale up automatically

Solution

1. Step 1: Understand readiness probe failure effect

   Readiness probe failure marks pod as not ready, so it stops receiving traffic.

2. Step 2: Differentiate from liveness probe effect

   Liveness probe failure triggers pod restart, readiness does not.

3. Final Answer:

   The pod will be marked as not ready and removed from service endpoints -> Option B

4. Quick Check:

   Readiness failure = pod not ready, no restart [OK]

Hint: Readiness failure removes pod from load balancer, no restart [OK]

Common Mistakes:

• Confusing readiness failure with pod restart
• Assuming pod scales automatically on probe failure
• Ignoring failureThreshold effect

4. A microservice has a liveness probe configured as an HTTP GET on /health. The service sometimes returns HTTP 500 during startup but is healthy afterward. What is the best fix to avoid unnecessary restarts?

medium

A. Increase initialDelaySeconds to allow startup time before probing

B. Change the probe to readiness probe instead of liveness probe

C. Remove the probe completely to avoid restarts

D. Set failureThreshold to 1 to detect failures faster

Solution

1. Step 1: Identify cause of restarts

   Liveness probe fails during startup because service returns HTTP 500 before ready.

2. Step 2: Adjust probe timing to avoid false failures

   Increasing initialDelaySeconds delays probe start, allowing service to become healthy first.

3. Final Answer:

   Increase initialDelaySeconds to allow startup time before probing -> Option A

4. Quick Check:

   Delay liveness probe start to avoid false failures [OK]

Hint: Delay liveness probe start to avoid false failure during startup [OK]

Common Mistakes:

• Removing probes which reduces reliability
• Confusing readiness and liveness probe roles
• Setting failureThreshold too low causing quick restarts

5. You have a microservice that takes time to initialize resources before it can serve requests. You want to ensure it is not restarted unnecessarily but also not receive traffic before ready. How should you configure liveness and readiness probes?

hard

A. Set liveness probe with a longer initialDelaySeconds and readiness probe to check resource initialization

B. Use only a liveness probe with a short periodSeconds to restart fast

C. Use only a readiness probe and no liveness probe

D. Set both probes to the same HTTP path and timing

Solution

1. Step 1: Prevent unnecessary restarts during initialization

   Set liveness probe initialDelaySeconds long enough to avoid restarting while initializing.

2. Step 2: Use readiness probe to block traffic until ready

   Readiness probe should check if resources are initialized before accepting traffic.

3. Final Answer:

   Set liveness probe with a longer initialDelaySeconds and readiness probe to check resource initialization -> Option A

4. Quick Check:

   Liveness delay + readiness check = safe startup [OK]

Hint: Delay liveness, readiness blocks traffic until ready [OK]

Common Mistakes:

• Using only one probe type causing traffic or restart issues
• Setting same path and timing for both probes
• Not delaying liveness probe causing premature restarts