Centralized logging (ELK stack) in Microservices - System Design Exercise

Design: Centralized Logging System using ELK Stack

Includes log collection, storage, search, and visualization. Excludes log generation and microservice internal logging implementation.

Functional Requirements

FR1: Collect logs from multiple microservices in real-time

FR2: Store logs centrally for easy search and analysis

FR3: Provide a dashboard for monitoring logs with filtering and alerting

FR4: Support log retention for at least 30 days

FR5: Handle at least 10,000 log events per second

FR6: Ensure logs are searchable with p99 query latency under 200ms

Non-Functional Requirements

NFR1: System must be highly available with 99.9% uptime

NFR2: Logs must be securely transmitted and stored

NFR3: The system should scale horizontally as log volume grows

NFR4: Minimal impact on microservices performance when sending logs

Think Before You Design

Questions to Ask

❓ Question 1

❓ Question 2

❓ Question 3

❓ Question 4

❓ Question 5

Key Components

Log shippers (e.g., Filebeat, Logstash)

Message queue or buffer for log ingestion

Elasticsearch cluster for storage and search

Kibana for visualization and dashboards

Security components like TLS and authentication

Design Patterns

Log aggregation

Event streaming with buffering

Indexing and search optimization

Horizontal scaling and sharding

Data retention and archival

Reference Architecture

Microservices --> Filebeat/Logstash --> Kafka (buffer) --> Elasticsearch Cluster --> Kibana Dashboard
                    |                     |                     |                     |
                    |                     |                     |                     |
                    +---------------------+---------------------+---------------------+

Components

Microservices

Any microservice framework

Generate application logs in structured format

Filebeat / Logstash

Elastic Beats / Logstash

Collect and forward logs from microservices reliably

Kafka

Apache Kafka

Buffer logs to handle spikes and decouple producers from consumers

Elasticsearch Cluster

Elasticsearch

Store, index, and provide fast search over logs

Kibana

Kibana

Visualize logs, create dashboards, and set alerts

Request Flow

1. Microservices generate logs and write to local files or stdout.

2. Filebeat agents installed on microservice hosts read logs and forward them to Kafka or Logstash.

3. Kafka buffers incoming logs to handle bursts and ensure durability.

4. Logstash consumes logs from Kafka, processes and transforms them if needed, then sends to Elasticsearch.

5. Elasticsearch indexes logs for fast search and stores them with retention policies.

6. Kibana connects to Elasticsearch to provide dashboards and alerting interfaces for users.

Database Schema

Elasticsearch stores logs as documents with fields: timestamp, service_name, log_level, message, trace_id, host, and custom tags. Indexes are created on timestamp and service_name for efficient querying.

Scaling Discussion

Bottlenecks

Log ingestion rate exceeding Kafka or Logstash capacity

Elasticsearch cluster storage and query performance limits

Network bandwidth between microservices and log shippers

Kibana dashboard performance with large datasets

Solutions

Scale Kafka brokers horizontally and partition topics for parallelism

Add Elasticsearch nodes and use sharding to distribute data

Use compression and batching in Filebeat to reduce network load

Implement index lifecycle management to archive or delete old logs

Optimize Kibana queries and use filters to limit data volume

Interview Tips

Time: Spend 10 minutes understanding requirements and clarifying scale. Use 20 minutes to design components and data flow. Reserve 10 minutes to discuss scaling and trade-offs. Use last 5 minutes for questions.

Explain how logs flow from microservices to Elasticsearch

Discuss buffering with Kafka to handle spikes

Highlight importance of indexing and search optimization

Mention security and data retention considerations

Describe scaling strategies for each component

Practice

1. What is the main purpose of the ELK stack in microservices architecture?

easy

A. To manage database transactions

B. To deploy microservices automatically

C. To collect, store, and visualize logs from multiple services in one place

D. To monitor network traffic between services

Solution

1. Step 1: Understand ELK stack components

   ELK stands for Elasticsearch (storage), Logstash (processing), and Kibana (visualization), all focused on logs.

2. Step 2: Identify ELK stack role in microservices

   It centralizes logs from many services to one place for easier monitoring and troubleshooting.

3. Final Answer:

   To collect, store, and visualize logs from multiple services in one place -> Option C

4. Quick Check:

   ELK stack = centralized logging [OK]

Hint: ELK = Elasticsearch + Logstash + Kibana for logs [OK]

Common Mistakes:

• Confusing ELK with deployment tools
• Thinking ELK manages databases
• Assuming ELK monitors network traffic

2. Which of the following is the correct Docker Compose service name for running Elasticsearch in an ELK stack?

easy

A. elasticsearch

B. kibana

C. logstash

D. filebeat

Solution

1. Step 1: Recall ELK stack components

   Elasticsearch stores logs, Logstash processes, Kibana visualizes, Filebeat ships logs.

2. Step 2: Identify correct service name in Docker Compose

   The service running Elasticsearch is named "elasticsearch" in Docker Compose files.

3. Final Answer:

   elasticsearch -> Option A

4. Quick Check:

   Elasticsearch service = elasticsearch [OK]

Hint: Elasticsearch service is named 'elasticsearch' in Docker Compose [OK]

Common Mistakes:

• Confusing Logstash or Kibana as Elasticsearch service
• Using 'filebeat' as ELK core service
• Misspelling service names

3. Given this Logstash configuration snippet:

input { beats { port => 5044 } } output { elasticsearch { hosts => ["http://elasticsearch:9200"] } }

What happens when Logstash receives logs on port 5044?

medium

A. Logs are discarded because port 5044 is incorrect

B. Logs are sent to Elasticsearch at http://elasticsearch:9200

C. Logs are visualized directly by Kibana

D. Logs are stored locally on Logstash server

Solution

1. Step 1: Analyze Logstash input configuration

   Logstash listens for logs from Beats agents on port 5044.

2. Step 2: Analyze Logstash output configuration

   Logs received are forwarded to Elasticsearch at the specified host and port.

3. Final Answer:

   Logs are sent to Elasticsearch at http://elasticsearch:9200 -> Option B

4. Quick Check:

   Logstash input port 5044 forwards logs to Elasticsearch [OK]

Hint: Logstash input port 5044 sends logs to Elasticsearch host [OK]

Common Mistakes:

• Assuming logs go directly to Kibana
• Thinking port 5044 is invalid
• Believing logs are stored locally on Logstash

4. You configured Logstash to receive logs on port 5044, but no logs appear in Elasticsearch. Which is the most likely cause?

medium

A. Docker Compose file is missing Kibana service

B. Kibana is not running

C. Logstash input port is set to 9200 instead of 5044

D. Elasticsearch service is down or unreachable

Solution

1. Step 1: Check connectivity between Logstash and Elasticsearch

   If Elasticsearch is down or unreachable, Logstash cannot send logs to it.

2. Step 2: Verify other options

   Kibana not running or missing does not stop logs from reaching Elasticsearch; wrong input port would prevent Logstash from receiving logs, not sending.

3. Final Answer:

   Elasticsearch service is down or unreachable -> Option D

4. Quick Check:

   Logs missing usually means Elasticsearch unreachable [OK]

Hint: Check Elasticsearch status if logs don't appear [OK]

Common Mistakes:

• Blaming Kibana for missing logs in Elasticsearch
• Confusing input port with Elasticsearch port
• Ignoring Elasticsearch service health

5. You want to add a new microservice that sends logs to the ELK stack using Filebeat. Which steps should you take to ensure logs appear in Kibana?

hard

A. Install Filebeat on the microservice host, configure it to send logs to Logstash on port 5044, and verify Elasticsearch and Kibana are running

B. Install Kibana on the microservice host and configure it to collect logs directly

C. Configure Elasticsearch to pull logs from the microservice host automatically

D. Run Logstash on the microservice host and send logs directly to Kibana

Solution

1. Step 1: Setup Filebeat on microservice host

   Filebeat collects logs locally and forwards them to Logstash on port 5044.

2. Step 2: Ensure ELK stack components are running

   Logstash processes logs, sends them to Elasticsearch, and Kibana visualizes them.

3. Final Answer:

   Install Filebeat on the microservice host, configure it to send logs to Logstash on port 5044, and verify Elasticsearch and Kibana are running -> Option A

4. Quick Check:

   Filebeat -> Logstash -> Elasticsearch -> Kibana [OK]

Hint: Filebeat sends logs to Logstash; Kibana visualizes them [OK]

Common Mistakes:

• Trying to send logs directly to Kibana
• Expecting Elasticsearch to pull logs automatically
• Running Logstash on microservice host unnecessarily