Why ELK stack provides observability in Elasticsearch - See It in Action

Why ELK stack provides observability

📖 Scenario: You work as a system administrator for a company that wants to monitor its web servers and applications. You want to collect logs, analyze them, and visualize the data to understand system health and performance.

🎯 Goal: Build a simple ELK stack setup to collect logs, search them, and create visualizations that help observe system behavior and detect issues.

📋 What You'll Learn

Create an Elasticsearch index with sample log data

Configure a filter to parse log entries

Use Kibana to create a visualization of error counts

Print a summary of why ELK stack helps with observability

💡 Why This Matters

🌍 Real World

Companies use ELK stack to monitor servers and applications in real time, helping them find and fix problems quickly.

💼 Career

Skills in ELK stack are valuable for roles like system administrator, DevOps engineer, and site reliability engineer.

1

Create sample log data in Elasticsearch

Create an Elasticsearch index called webserver-logs with these exact documents: {"timestamp": "2024-06-01T12:00:00Z", "level": "INFO", "message": "Server started"}, {"timestamp": "2024-06-01T12:05:00Z", "level": "ERROR", "message": "Database connection failed"}, and {"timestamp": "2024-06-01T12:10:00Z", "level": "WARN", "message": "High memory usage"}.

Elasticsearch

# Create the index and add the three documents with exact fields and values
# Your code here

Hint

Use Elasticsearch PUT requests to add documents to the webserver-logs index with the exact fields and values.

2

Add a filter to parse log levels

Create a Logstash filter configuration that parses the level field from the logs and tags errors with error_tag. Use the exact filter syntax: if [level] == "ERROR" { mutate { add_tag => ["error_tag"] } }.

Elasticsearch

PUT /webserver-logs/_doc/1
{
  "timestamp": "2024-06-01T12:00:00Z",
  "level": "INFO",
  "message": "Server started"
}

PUT /webserver-logs/_doc/2
{
  "timestamp": "2024-06-01T12:05:00Z",
  "level": "ERROR",
  "message": "Database connection failed"
}

PUT /webserver-logs/_doc/3
{
  "timestamp": "2024-06-01T12:10:00Z",
  "level": "WARN",
  "message": "High memory usage"
}

# Add the Logstash filter configuration below
# Your code here

Hint

Use Logstash filter syntax to check if level equals "ERROR" and add the tag "error_tag".

3

Create a Kibana visualization for error counts

Write a Kibana query that counts documents with the tag error_tag in the webserver-logs index. Use the exact query: { "query": { "term": { "tags": "error_tag" } } }.

Elasticsearch

PUT /webserver-logs/_doc/1
{
  "timestamp": "2024-06-01T12:00:00Z",
  "level": "INFO",
  "message": "Server started"
}

PUT /webserver-logs/_doc/2
{
  "timestamp": "2024-06-01T12:05:00Z",
  "level": "ERROR",
  "message": "Database connection failed"
}

PUT /webserver-logs/_doc/3
{
  "timestamp": "2024-06-01T12:10:00Z",
  "level": "WARN",
  "message": "High memory usage"
}

filter {
  if [level] == "ERROR" {
    mutate {
      add_tag => ["error_tag"]
    }
  }
}

# Add the Kibana query below
# Your code here

Hint

Use a Kibana JSON query to filter documents where the tags field contains "error_tag".

4

Print why ELK stack provides observability

Write a print statement that outputs exactly: "ELK stack provides observability by collecting, parsing, and visualizing logs to monitor system health and detect issues."

Elasticsearch

PUT /webserver-logs/_doc/1
{
  "timestamp": "2024-06-01T12:00:00Z",
  "level": "INFO",
  "message": "Server started"
}

PUT /webserver-logs/_doc/2
{
  "timestamp": "2024-06-01T12:05:00Z",
  "level": "ERROR",
  "message": "Database connection failed"
}

PUT /webserver-logs/_doc/3
{
  "timestamp": "2024-06-01T12:10:00Z",
  "level": "WARN",
  "message": "High memory usage"
}

filter {
  if [level] == "ERROR" {
    mutate {
      add_tag => ["error_tag"]
    }
  }
}

{
  "query": {
    "term": {
      "tags": "error_tag"
    }
  }
}

# Print the explanation below
# Your code here

Hint

Use print() with the exact sentence inside quotes.

Practice

1. What is the main reason the ELK stack provides observability in systems?
ELK = Elasticsearch + Logstash + Kibana

easy

A. It collects, stores, and visualizes data to understand system behavior

B. It only stores data without visualization

C. It only visualizes data without collecting it

D. It replaces all system monitoring tools automatically

Solution

1. Step 1: Understand ELK components roles

   Elasticsearch stores data, Logstash collects and processes data, Kibana visualizes data.

2. Step 2: Connect roles to observability

   Combining these lets you see and understand system behavior clearly.

3. Final Answer:

   It collects, stores, and visualizes data to understand system behavior -> Option A

4. Quick Check:

   Observability = Collect + Store + Visualize [OK]

Hint: Remember ELK = Collect + Store + Visualize for observability [OK]

Common Mistakes:

• Thinking ELK only stores data
• Assuming ELK only visualizes data
• Believing ELK replaces all monitoring tools automatically

2. Which syntax correctly shows the ELK stack components working together for observability?

easy

A. Logstash -> Elasticsearch -> Kibana

B. Kibana -> Logstash -> Elasticsearch

C. Elasticsearch -> Kibana -> Logstash

D. Logstash -> Kibana -> Elasticsearch

Solution

1. Step 1: Identify data flow in ELK

   Logstash collects and processes data first, then sends it to Elasticsearch for storage.

2. Step 2: Visualize data with Kibana

   Kibana reads data from Elasticsearch to create visual dashboards.

3. Final Answer:

   Logstash -> Elasticsearch -> Kibana -> Option A

4. Quick Check:

   Data flow = Logstash to Elasticsearch to Kibana [OK]

Hint: Data flows Logstash -> Elasticsearch -> Kibana [OK]

Common Mistakes:

• Mixing order of components
• Thinking Kibana collects data
• Assuming Elasticsearch visualizes data

3. Given the ELK stack setup, what will Kibana display if Logstash collects logs and Elasticsearch stores them correctly?

medium

A. Only error messages without context

B. Raw logs without any visualization

C. Visual dashboards showing system logs and metrics

D. No data because Kibana cannot access Elasticsearch

Solution

1. Step 1: Understand Kibana's role

   Kibana reads data from Elasticsearch and creates visual dashboards.

2. Step 2: Consider data flow correctness

   If Logstash collects logs and Elasticsearch stores them, Kibana can visualize them properly.

3. Final Answer:

   Visual dashboards showing system logs and metrics -> Option C

4. Quick Check:

   Kibana visualizes stored data [OK]

Hint: Kibana shows dashboards if data is stored correctly [OK]

Common Mistakes:

• Thinking Kibana shows raw logs only
• Assuming Kibana cannot access Elasticsearch
• Believing Kibana shows only errors

4. You set up ELK stack but Kibana shows no data. What is the most likely error in your setup?

medium

A. Elasticsearch is visualizing data incorrectly

B. Kibana is collecting data instead of visualizing

C. Logstash is visualizing data directly

D. Logstash is not sending data to Elasticsearch

Solution

1. Step 1: Identify data flow problem

   If Kibana shows no data, likely Elasticsearch has no data to show.

2. Step 2: Check Logstash role

   Logstash must send data to Elasticsearch; if it doesn't, Elasticsearch stays empty.

3. Final Answer:

   Logstash is not sending data to Elasticsearch -> Option D

4. Quick Check:

   No data in Kibana means no data in Elasticsearch [OK]

Hint: Check Logstash to Elasticsearch connection first [OK]

Common Mistakes:

• Thinking Kibana collects data
• Assuming Elasticsearch visualizes data
• Believing Logstash visualizes data

5. How does the ELK stack help a team quickly find and fix issues in a complex system?

hard

A. By automatically fixing bugs without human input

B. By collecting logs, storing them centrally, and visualizing patterns and errors

C. By replacing all system components with ELK tools

D. By only storing data without any analysis or visualization

Solution

1. Step 1: Understand ELK's observability role

   ELK collects logs, stores them centrally, and visualizes data to reveal system behavior.

2. Step 2: Connect observability to issue resolution

   Visualizing patterns and errors helps teams quickly spot and fix problems.

3. Final Answer:

   By collecting logs, storing them centrally, and visualizing patterns and errors -> Option B

4. Quick Check:

   Observability = Collect + Store + Visualize for quick fixes [OK]

Hint: Observability helps find and fix issues fast [OK]

Common Mistakes:

• Thinking ELK fixes bugs automatically
• Assuming ELK replaces all system parts
• Believing storing data alone solves issues