Why ELK stack provides observability in Elasticsearch - See It in Action

Why ELK stack provides observability

📖 Scenario: You work as a system administrator for a company that wants to monitor its web servers and applications. You want to collect logs, analyze them, and visualize the data to understand system health and performance.

🎯 Goal: Build a simple ELK stack setup to collect logs, search them, and create visualizations that help observe system behavior and detect issues.

📋 What You'll Learn

Create an Elasticsearch index with sample log data

Configure a filter to parse log entries

Use Kibana to create a visualization of error counts

Print a summary of why ELK stack helps with observability

💡 Why This Matters

🌍 Real World

Companies use ELK stack to monitor servers and applications in real time, helping them find and fix problems quickly.

💼 Career

Skills in ELK stack are valuable for roles like system administrator, DevOps engineer, and site reliability engineer.

1

Create sample log data in Elasticsearch

Create an Elasticsearch index called webserver-logs with these exact documents: {"timestamp": "2024-06-01T12:00:00Z", "level": "INFO", "message": "Server started"}, {"timestamp": "2024-06-01T12:05:00Z", "level": "ERROR", "message": "Database connection failed"}, and {"timestamp": "2024-06-01T12:10:00Z", "level": "WARN", "message": "High memory usage"}.

Elasticsearch

# Create the index and add the three documents with exact fields and values
# Your code here

Need a hint?

Use Elasticsearch PUT requests to add documents to the webserver-logs index with the exact fields and values.

2

Add a filter to parse log levels

Create a Logstash filter configuration that parses the level field from the logs and tags errors with error_tag. Use the exact filter syntax: if [level] == "ERROR" { mutate { add_tag => ["error_tag"] } }.

Elasticsearch

PUT /webserver-logs/_doc/1
{
  "timestamp": "2024-06-01T12:00:00Z",
  "level": "INFO",
  "message": "Server started"
}

PUT /webserver-logs/_doc/2
{
  "timestamp": "2024-06-01T12:05:00Z",
  "level": "ERROR",
  "message": "Database connection failed"
}

PUT /webserver-logs/_doc/3
{
  "timestamp": "2024-06-01T12:10:00Z",
  "level": "WARN",
  "message": "High memory usage"
}

# Add the Logstash filter configuration below
# Your code here

Need a hint?

Use Logstash filter syntax to check if level equals "ERROR" and add the tag "error_tag".

3

Create a Kibana visualization for error counts

Write a Kibana query that counts documents with the tag error_tag in the webserver-logs index. Use the exact query: { "query": { "term": { "tags": "error_tag" } } }.

Elasticsearch

PUT /webserver-logs/_doc/1
{
  "timestamp": "2024-06-01T12:00:00Z",
  "level": "INFO",
  "message": "Server started"
}

PUT /webserver-logs/_doc/2
{
  "timestamp": "2024-06-01T12:05:00Z",
  "level": "ERROR",
  "message": "Database connection failed"
}

PUT /webserver-logs/_doc/3
{
  "timestamp": "2024-06-01T12:10:00Z",
  "level": "WARN",
  "message": "High memory usage"
}

filter {
  if [level] == "ERROR" {
    mutate {
      add_tag => ["error_tag"]
    }
  }
}

# Add the Kibana query below
# Your code here

Need a hint?

Use a Kibana JSON query to filter documents where the tags field contains "error_tag".

4

Print why ELK stack provides observability

Write a print statement that outputs exactly: "ELK stack provides observability by collecting, parsing, and visualizing logs to monitor system health and detect issues."

Elasticsearch

PUT /webserver-logs/_doc/1
{
  "timestamp": "2024-06-01T12:00:00Z",
  "level": "INFO",
  "message": "Server started"
}

PUT /webserver-logs/_doc/2
{
  "timestamp": "2024-06-01T12:05:00Z",
  "level": "ERROR",
  "message": "Database connection failed"
}

PUT /webserver-logs/_doc/3
{
  "timestamp": "2024-06-01T12:10:00Z",
  "level": "WARN",
  "message": "High memory usage"
}

filter {
  if [level] == "ERROR" {
    mutate {
      add_tag => ["error_tag"]
    }
  }
}

{
  "query": {
    "term": {
      "tags": "error_tag"
    }
  }
}

# Print the explanation below
# Your code here

Need a hint?

Use print() with the exact sentence inside quotes.