Bird
Raised Fist0
Microservicessystem_design~12 mins

Chaos engineering basics in Microservices - Architecture Diagram

Choose your learning style10 modes available
LearnWhyDeepArchPracticeChallengeDesignRecallScale

Start learning this pattern below

Jump into concepts and practice - no test required

or
Recommended
Test this pattern10 questions across easy, medium, and hard to know if this pattern is strong
System Overview - Chaos engineering basics

Chaos engineering is about testing how a system behaves when parts fail unexpectedly. It helps find weaknesses before real problems happen. The system is designed with microservices that communicate through APIs, with monitoring and fallback mechanisms to keep services running during failures.

Architecture Diagram
          +------------+          +----------------+
          |            |          |                |
User --->| Load       |--------->| API Gateway    | 
          | Balancer   |          |                |
          +------------+          +----------------+
                                       |      
             +-------------------------+-------------------------+
             |                         |                         |
      +--------------+          +--------------+          +--------------+
      | Service A    |          | Service B    |          | Service C    |
      +--------------+          +--------------+          +--------------+
             |                         |                         |
      +--------------+          +--------------+          +--------------+
      | Database A   |          | Database B   |          | Database C   |
      +--------------+          +--------------+          +--------------+
             |                         |                         |
      +--------------+          +--------------+          +--------------+
      | Cache A      |          | Cache B      |          | Cache C      |
      +--------------+          +--------------+          +--------------+

Monitoring & Chaos Injector Service connected to API Gateway and Services
Components
User
client
Sends requests to the system
Load Balancer
load_balancer
Distributes incoming user requests evenly to API Gateway instances
API Gateway
api_gateway
Routes requests to appropriate microservices and enforces security
Service A
service
Handles specific business logic and communicates with Database A and Cache A
Service B
service
Handles another business domain and communicates with Database B and Cache B
Service C
service
Handles additional business logic and communicates with Database C and Cache C
Database A
database
Stores persistent data for Service A
Database B
database
Stores persistent data for Service B
Database C
database
Stores persistent data for Service C
Cache A
cache
Speeds up data access for Service A by caching frequent queries
Cache B
cache
Speeds up data access for Service B by caching frequent queries
Cache C
cache
Speeds up data access for Service C by caching frequent queries
Monitoring & Chaos Injector Service
service
Monitors system health and injects failures to test system resilience
Request Flow - 13 Hops
User → Load Balancer
Load Balancer → API Gateway
API Gateway → Service A
Service A → Cache A
Cache A → Service A
Service A → Database A
Database A → Service A
Service A → Cache A
Service A → API Gateway
API Gateway → Load Balancer
Load Balancer → User
Monitoring & Chaos Injector Service → Service B
Service B → Monitoring & Chaos Injector Service
Failure Scenario
Component Fails:Database B
Impact:Service B cannot read or write persistent data, causing errors or degraded functionality. Cache B may serve stale data but cannot update.
Mitigation:System detects failure via monitoring. Traffic to Service B can be rerouted or degraded gracefully. Database replication or failover can restore availability.
Architecture Quiz - 3 Questions
Test your understanding
Which component is responsible for distributing user requests evenly to prevent overload?
AAPI Gateway
BLoad Balancer
CCache
DDatabase
Design Principle
This architecture demonstrates how chaos engineering integrates failure injection and monitoring into a microservices system. It uses caching to improve performance and load balancing to distribute traffic. The design ensures resilience by detecting failures early and enabling graceful degradation or recovery.

Practice

(1/5)
1. What is the main goal of chaos engineering in microservices?
easy
A. To reduce the number of developers needed
B. To increase the number of microservices in a system
C. To find and fix weaknesses before real failures occur
D. To speed up the deployment process

Solution

  1. Step 1: Understand chaos engineering purpose

    Chaos engineering is about testing systems by intentionally causing failures to find weaknesses.

  2. Step 2: Identify the main goal

    The goal is to find and fix weaknesses before they cause real problems in production.

  3. Final Answer:

    To find and fix weaknesses before real failures occur -> Option C

  4. Quick Check:

    Chaos engineering goal = Find and fix weaknesses [OK]
Hint: Chaos engineering tests failures to improve system stability [OK]
Common Mistakes:
  • Thinking chaos engineering increases microservices count
  • Confusing chaos engineering with deployment speedup
  • Assuming chaos engineering reduces developer count
2. Which of the following is a correct way to start chaos engineering experiments?
easy
A. Start with complex multi-service failures immediately
B. Begin with simple, controlled failure tests
C. Run chaos tests only after a system crash
D. Avoid monitoring during chaos experiments

Solution

  1. Step 1: Review best practice for chaos experiments

    Best practice is to start small with simple, controlled failures to understand system behavior.

  2. Step 2: Identify the correct starting approach

    Starting with simple tests helps safely learn and improve system resilience gradually.

  3. Final Answer:

    Begin with simple, controlled failure tests -> Option B

  4. Quick Check:

    Start chaos with simple tests = Begin with simple, controlled failure tests [OK]
Hint: Start chaos tests simple and controlled, not complex [OK]
Common Mistakes:
  • Starting with complex failures too soon
  • Running chaos only after failures happen
  • Ignoring monitoring during tests
3. Consider a microservice system where a chaos experiment randomly kills one instance every 5 minutes. What is the expected immediate effect on system availability?
medium
A. System availability remains stable if redundancy exists
B. System availability drops to zero immediately
C. System crashes permanently after first kill
D. System automatically scales down instances

Solution

  1. Step 1: Analyze the chaos experiment impact

    Killing one instance every 5 minutes tests resilience but does not remove all instances.

  2. Step 2: Consider system redundancy

    If the system has redundant instances, killing one does not reduce availability immediately.

  3. Final Answer:

    System availability remains stable if redundancy exists -> Option A

  4. Quick Check:

    Redundancy keeps availability stable during chaos [OK]
Hint: Redundancy keeps system available despite instance failures [OK]
Common Mistakes:
  • Assuming system crashes immediately after one instance killed
  • Thinking availability drops to zero instantly
  • Believing system scales down automatically
4. A chaos experiment script intended to shut down a microservice instance sometimes fails silently without stopping the instance. What is the most likely cause?
medium
A. The network is too fast for the script
B. The microservice is designed to never stop
C. The chaos experiment is running on a different system
D. The script lacks proper error handling and logging

Solution

  1. Step 1: Identify why script fails silently

    Silent failures usually happen when errors are not caught or logged properly.

  2. Step 2: Evaluate other options

    Microservices can be stopped; network speed does not cause silent failure; running on different system would cause errors, not silent failure.

  3. Final Answer:

    The script lacks proper error handling and logging -> Option D

  4. Quick Check:

    Silent failure = Missing error handling [OK]
Hint: Check error handling if chaos script fails silently [OK]
Common Mistakes:
  • Assuming microservice cannot be stopped
  • Blaming network speed for silent failure
  • Ignoring script environment mismatch
5. You want to design a chaos engineering experiment to test how your microservices handle database latency spikes. Which approach best fits this goal?
hard
A. Inject artificial latency into database calls during tests
B. Disable monitoring tools to avoid false alerts
C. Increase the number of database replicas without testing
D. Randomly kill microservice instances during peak hours

Solution

  1. Step 1: Understand the goal of testing database latency spikes

    The goal is to see how microservices behave when database responses are slow.

  2. Step 2: Choose the best chaos experiment approach

    Injecting artificial latency simulates slow database calls directly, matching the goal.

  3. Step 3: Evaluate other options

    Killing instances tests availability, not latency; increasing replicas without testing doesn't simulate latency; disabling monitoring hides important data.

  4. Final Answer:

    Inject artificial latency into database calls during tests -> Option A

  5. Quick Check:

    Test latency by injecting delays = Inject artificial latency into database calls during tests [OK]
Hint: Inject delays to test latency, not kill instances [OK]
Common Mistakes:
  • Confusing instance failure with latency testing
  • Adding replicas without testing effects
  • Turning off monitoring during chaos