Centralized logging (ELK stack) in Microservices - Architecture Diagram

System Overview - Centralized logging (ELK stack)

This system collects logs from multiple microservices into a central place for easy searching and analysis. It uses the ELK stack: Elasticsearch for storing logs, Logstash for processing, and Kibana for visualization. The system must handle high log volume and provide fast search.

Architecture Diagram

User
  |
  v
+----------------+
| Microservices  |
+----------------+
       |
       v
+----------------+
| Logstash       |
| (Log Processor)|
+----------------+
       |
       v
+----------------+
| Elasticsearch  |
| (Log Storage)  |
+----------------+
       |
       v
+----------------+
| Kibana         |
| (Visualization)|
+----------------+

Components

Microservices

service

Generate application logs and send them to Logstash

Logstash

service

Collect, parse, and transform logs from microservices before sending to Elasticsearch

Elasticsearch

database

Store and index logs for fast search and retrieval

Kibana

service

Provide a web interface to visualize and query logs stored in Elasticsearch

Request Flow - 5 Hops

Microservices → Logstash

Logstash → Elasticsearch

User → Kibana

Kibana → Elasticsearch

Elasticsearch → Kibana

Failure Scenario

Component Fails:Elasticsearch

Impact:Logs cannot be stored or searched; Kibana cannot retrieve data; system loses visibility into logs

Mitigation:Use Elasticsearch cluster with replication and failover; cache recent logs in Logstash or external storage temporarily

Architecture Quiz - 3 Questions

Which component is responsible for parsing and transforming logs before storage?

A Logstash

B Kibana

C Elasticsearch

D Microservices

Design Principle

This architecture centralizes logs from many microservices into one searchable system. It separates concerns: Logstash processes logs, Elasticsearch stores and indexes them, and Kibana visualizes them. This design scales well and supports fast log analysis.