Docker Compose for ELK Stack: Setup and Usage Guide

Use a docker-compose.yml file to define services for Elasticsearch, Logstash, and Kibana with proper ports and volumes. Running docker-compose up will start the ELK stack containers together, enabling easy log management and visualization.

Syntax

A docker-compose.yml file defines multiple services in one place. Each service has a name and configuration like image, ports, volumes, and environment variables.

For ELK stack, you define three services: elasticsearch, logstash, and kibana. You specify the Docker image for each, map ports to access them, and set environment variables for configuration.

yaml

version: '3.8'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.8.2
    environment:
      - discovery.type=single-node
    ports:
      - 9200:9200
    volumes:
      - esdata:/usr/share/elasticsearch/data

  logstash:
    image: docker.elastic.co/logstash/logstash:8.8.2
    ports:
      - 5044:5044
    volumes:
      - ./logstash/config:/usr/share/logstash/config
      - ./logstash/pipeline:/usr/share/logstash/pipeline

  kibana:
    image: docker.elastic.co/kibana/kibana:8.8.2
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch

volumes:
  esdata: {}

Example

This example docker-compose.yml file sets up the ELK stack with Elasticsearch running on port 9200, Logstash on 5044, and Kibana on 5601. It uses official Elastic images and configures Elasticsearch as a single-node cluster.

Run docker-compose up in the directory with this file to start all services. You can then access Kibana at http://localhost:5601 to visualize logs.

yaml

version: '3.8'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.8.2
    environment:
      - discovery.type=single-node
    ports:
      - 9200:9200
    volumes:
      - esdata:/usr/share/elasticsearch/data

  logstash:
    image: docker.elastic.co/logstash/logstash:8.8.2
    ports:
      - 5044:5044
    volumes:
      - ./logstash/config:/usr/share/logstash/config
      - ./logstash/pipeline:/usr/share/logstash/pipeline

  kibana:
    image: docker.elastic.co/kibana/kibana:8.8.2
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch

volumes:
  esdata: {}

Output

Creating network "elk_default" with the default driver
Creating volume "elk_esdata" with default driver
Creating elk_elasticsearch_1 ... done
Creating elk_logstash_1      ... done
Creating elk_kibana_1        ... done

Starting ELK stack containers. Access Kibana at http://localhost:5601

Common Pitfalls

• Not setting discovery.type=single-node in Elasticsearch: This causes Elasticsearch to wait for a cluster and never start in single-node mode.
• Port conflicts: Make sure ports 9200, 5044, and 5601 are free on your machine.
• Missing volumes for data persistence: Without volumes, Elasticsearch data is lost when containers stop.
• Incorrect Logstash pipeline configuration paths: Ensure your local ./logstash/config and ./logstash/pipeline folders exist and have valid config files.

yaml

version: '3.8'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.8.2
    # Missing discovery.type causes startup failure
    ports:
      - 9200:9200

  kibana:
    image: docker.elastic.co/kibana/kibana:8.8.2
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch

volumes:
  esdata: {}

Quick Reference

• Elasticsearch: Runs on port 9200, needs discovery.type=single-node for single-node setup.
• Logstash: Listens on port 5044, requires pipeline config files mounted.
• Kibana: Runs on port 5601, depends on Elasticsearch service.
• Volumes: Use named volumes for Elasticsearch data persistence.
• Start command: Run docker-compose up to launch all services.

Key Takeaways

Use a docker-compose.yml file to define Elasticsearch, Logstash, and Kibana services together.

Set discovery.type=single-node in Elasticsearch environment to run a single-node cluster.

Map ports 9200, 5044, and 5601 to access Elasticsearch, Logstash, and Kibana respectively.

Mount volumes for Elasticsearch data to keep logs persistent across container restarts.

Ensure Logstash config and pipeline files exist locally and are correctly mounted.