Bird
Raised Fist0
Microservicessystem_design~25 mins

Chaos engineering basics in Microservices - System Design Exercise

Choose your learning style10 modes available
LearnWhyDeepArchPracticeChallengeDesignRecallScale

Start learning this pattern below

Jump into concepts and practice - no test required

or
Recommended
Test this pattern10 questions across easy, medium, and hard to know if this pattern is strong
Design: Chaos Engineering Platform for Microservices
Design the chaos engineering platform components and integration with microservices. Out of scope: detailed microservice implementation or business logic.
Functional Requirements
FR1: Inject failures like latency, errors, and service crashes into microservices
FR2: Monitor system behavior and detect resilience issues automatically
FR3: Support scheduling and targeting specific microservices or endpoints
FR4: Provide dashboards for real-time metrics and failure impact visualization
FR5: Allow safe rollback and stop of experiments to avoid production damage
Non-Functional Requirements
NFR1: Handle up to 100 microservices in production
NFR2: Minimal impact on normal system latency (p99 < 200ms overhead)
NFR3: Availability target 99.9% uptime for the chaos platform itself
NFR4: Experiments must be isolated and reversible
NFR5: Secure access control for who can run chaos experiments
Think Before You Design
Questions to Ask
❓ Question 1
❓ Question 2
❓ Question 3
❓ Question 4
❓ Question 5
Key Components
Failure injection agents or sidecars in microservices
Central chaos control service with API and UI
Metrics collection and monitoring system
Experiment scheduler and rollback mechanism
Authentication and authorization for experiment control
Design Patterns
Circuit breaker pattern for resilience
Bulkhead isolation to limit failure blast radius
Canary deployments for safe testing
Event-driven monitoring and alerting
Feature flags to enable/disable chaos experiments
Reference Architecture
                    +-----------------------+
                    | Chaos Control Service  |
                    |  - API & UI           |
                    |  - Scheduler          |
                    |  - Auth & Rollback    |
                    +-----------+-----------+
                                |
                                v
       +------------------------+------------------------+
       | Failure Injection Agents (sidecars) in each    |
       | microservice instance                            |
       +------------------------+------------------------+
                                |
                                v
                    +-----------------------+
                    | Monitoring & Metrics   |
                    |  - Collect logs       |
                    |  - Track latency      |
                    |  - Alerting system    |
                    +-----------------------+
Components
Chaos Control Service
Node.js/Go REST API with React UI
Central interface to create, schedule, monitor, and rollback chaos experiments
Failure Injection Agents
Sidecar containers or middleware in microservices
Inject faults like latency, errors, or crashes into targeted microservices
Monitoring & Metrics System
Prometheus + Grafana + Alertmanager
Collect metrics and logs to observe system behavior during chaos experiments
Authentication & Authorization
OAuth2 / RBAC
Control who can run or stop chaos experiments to ensure security
Experiment Scheduler
Cron jobs or Kubernetes CronJobs
Automate running chaos experiments at defined times or intervals
Request Flow
1. User logs into Chaos Control Service UI with secure credentials.
2. User creates a chaos experiment specifying target microservices and failure types.
3. Scheduler triggers the experiment at the scheduled time.
4. Chaos Control Service sends commands to Failure Injection Agents in targeted microservices.
5. Agents inject faults (e.g., add latency, return errors) as instructed.
6. Monitoring system collects metrics and logs from microservices during experiment.
7. Chaos Control Service displays real-time impact on dashboard.
8. User can stop or rollback the experiment via the control service.
9. Agents revert injected faults to restore normal service behavior.
Database Schema
Entities: - User (id, username, password_hash, role) - Experiment (id, name, description, target_services, failure_types, schedule, status, created_at, updated_at) - Service (id, name, version, metadata) - ExperimentLog (id, experiment_id, timestamp, metric_name, metric_value) Relationships: - User creates many Experiments (1:N) - Experiment targets many Services (N:N) - Experiment has many ExperimentLogs (1:N)
Scaling Discussion
Bottlenecks
Chaos Control Service becomes overloaded with many concurrent experiments
Failure Injection Agents add latency overhead to microservices
Monitoring system overwhelmed by high volume of metrics during experiments
Security risks if unauthorized users access chaos controls
Rollback delays causing prolonged system instability
Solutions
Scale Chaos Control Service horizontally with load balancing and stateless design
Optimize agents to minimize latency; use sampling or throttling of fault injection
Use scalable monitoring backends like Thanos or Cortex for Prometheus metrics
Implement strict RBAC and audit logging for experiment control access
Automate rollback triggers based on anomaly detection to reduce manual delays
Interview Tips
Time: Spend 10 minutes understanding requirements and clarifying scope, 20 minutes designing components and data flow, 10 minutes discussing scaling and security, 5 minutes summarizing.
Explain importance of safe, reversible chaos experiments in production
Describe how failure injection agents work and integrate with microservices
Highlight monitoring and metrics collection for impact analysis
Discuss security controls to prevent misuse
Address scaling challenges and solutions for large microservice environments

Practice

(1/5)
1. What is the main goal of chaos engineering in microservices?
easy
A. To reduce the number of developers needed
B. To increase the number of microservices in a system
C. To find and fix weaknesses before real failures occur
D. To speed up the deployment process

Solution

  1. Step 1: Understand chaos engineering purpose

    Chaos engineering is about testing systems by intentionally causing failures to find weaknesses.

  2. Step 2: Identify the main goal

    The goal is to find and fix weaknesses before they cause real problems in production.

  3. Final Answer:

    To find and fix weaknesses before real failures occur -> Option C

  4. Quick Check:

    Chaos engineering goal = Find and fix weaknesses [OK]
Hint: Chaos engineering tests failures to improve system stability [OK]
Common Mistakes:
  • Thinking chaos engineering increases microservices count
  • Confusing chaos engineering with deployment speedup
  • Assuming chaos engineering reduces developer count
2. Which of the following is a correct way to start chaos engineering experiments?
easy
A. Start with complex multi-service failures immediately
B. Begin with simple, controlled failure tests
C. Run chaos tests only after a system crash
D. Avoid monitoring during chaos experiments

Solution

  1. Step 1: Review best practice for chaos experiments

    Best practice is to start small with simple, controlled failures to understand system behavior.

  2. Step 2: Identify the correct starting approach

    Starting with simple tests helps safely learn and improve system resilience gradually.

  3. Final Answer:

    Begin with simple, controlled failure tests -> Option B

  4. Quick Check:

    Start chaos with simple tests = Begin with simple, controlled failure tests [OK]
Hint: Start chaos tests simple and controlled, not complex [OK]
Common Mistakes:
  • Starting with complex failures too soon
  • Running chaos only after failures happen
  • Ignoring monitoring during tests
3. Consider a microservice system where a chaos experiment randomly kills one instance every 5 minutes. What is the expected immediate effect on system availability?
medium
A. System availability remains stable if redundancy exists
B. System availability drops to zero immediately
C. System crashes permanently after first kill
D. System automatically scales down instances

Solution

  1. Step 1: Analyze the chaos experiment impact

    Killing one instance every 5 minutes tests resilience but does not remove all instances.

  2. Step 2: Consider system redundancy

    If the system has redundant instances, killing one does not reduce availability immediately.

  3. Final Answer:

    System availability remains stable if redundancy exists -> Option A

  4. Quick Check:

    Redundancy keeps availability stable during chaos [OK]
Hint: Redundancy keeps system available despite instance failures [OK]
Common Mistakes:
  • Assuming system crashes immediately after one instance killed
  • Thinking availability drops to zero instantly
  • Believing system scales down automatically
4. A chaos experiment script intended to shut down a microservice instance sometimes fails silently without stopping the instance. What is the most likely cause?
medium
A. The network is too fast for the script
B. The microservice is designed to never stop
C. The chaos experiment is running on a different system
D. The script lacks proper error handling and logging

Solution

  1. Step 1: Identify why script fails silently

    Silent failures usually happen when errors are not caught or logged properly.

  2. Step 2: Evaluate other options

    Microservices can be stopped; network speed does not cause silent failure; running on different system would cause errors, not silent failure.

  3. Final Answer:

    The script lacks proper error handling and logging -> Option D

  4. Quick Check:

    Silent failure = Missing error handling [OK]
Hint: Check error handling if chaos script fails silently [OK]
Common Mistakes:
  • Assuming microservice cannot be stopped
  • Blaming network speed for silent failure
  • Ignoring script environment mismatch
5. You want to design a chaos engineering experiment to test how your microservices handle database latency spikes. Which approach best fits this goal?
hard
A. Inject artificial latency into database calls during tests
B. Disable monitoring tools to avoid false alerts
C. Increase the number of database replicas without testing
D. Randomly kill microservice instances during peak hours

Solution

  1. Step 1: Understand the goal of testing database latency spikes

    The goal is to see how microservices behave when database responses are slow.

  2. Step 2: Choose the best chaos experiment approach

    Injecting artificial latency simulates slow database calls directly, matching the goal.

  3. Step 3: Evaluate other options

    Killing instances tests availability, not latency; increasing replicas without testing doesn't simulate latency; disabling monitoring hides important data.

  4. Final Answer:

    Inject artificial latency into database calls during tests -> Option A

  5. Quick Check:

    Test latency by injecting delays = Inject artificial latency into database calls during tests [OK]
Hint: Inject delays to test latency, not kill instances [OK]
Common Mistakes:
  • Confusing instance failure with latency testing
  • Adding replicas without testing effects
  • Turning off monitoring during chaos