0
0
Dockerdevops~5 mins

Centralized logging setup in Docker - Commands & Configuration

Choose your learning style9 modes available
LearnWhyDeepVisualTryChallengeProjectRecallTime
Introduction
When you run many containers, it is hard to find logs from each one separately. Centralized logging collects all logs in one place so you can see and search them easily.
When you want to see logs from multiple containers in one dashboard.
When you need to keep logs safe even if containers stop or restart.
When you want to analyze logs to find errors or performance issues.
When you want to share logs with your team without accessing each server.
When you want to keep logs for a long time without filling container storage.
Config File - docker-compose.yml
docker-compose.yml
version: '3.8'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.6.3
    environment:
      - discovery.type=single-node
    ports:
      - "9200:9200"
  kibana:
    image: docker.elastic.co/kibana/kibana:8.6.3
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
  fluentd:
    image: fluent/fluentd:v1.15-debian-1
    volumes:
      - ./fluentd.conf:/fluentd/etc/fluentd.conf
    ports:
      - "24224:24224"
      - "24224:24224/udp"
  my-app:
    image: nginx:1.23
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: my-app

This file sets up four services:

  • Elasticsearch: stores logs and allows searching.
  • Kibana: web interface to view and analyze logs.
  • Fluentd: collects logs from containers and sends to Elasticsearch.
  • my-app: example app container sending logs to Fluentd.

The logging section in my-app tells Docker to send logs to Fluentd on port 24224.

Commands
Start all services in the background: Elasticsearch, Kibana, Fluentd, and the example app.
Terminal
docker-compose up -d
Expected OutputExpected
Creating network "default" with the default driver Creating volume "default_elasticsearch_data" with default driver Creating default_elasticsearch_1 ... done Creating default_kibana_1 ... done Creating default_fluentd_1 ... done Creating default_my-app_1 ... done
→
-d - Run containers in detached mode (in background)
Check that all containers are running properly after starting them.
Terminal
docker-compose ps
Expected OutputExpected
Name Command State Ports default_elasticsearch_1 /bin/tini -- /usr/local/bin/docker-entrypoint.sh elasticsearch Up 0.0.0.0:9200->9200/tcp default_kibana_1 /bin/tini -- /usr/local/bin/kibana-docker Up 0.0.0.0:5601->5601/tcp default_fluentd_1 fluentd -c /fluentd/etc/fluentd.conf Up 0.0.0.0:24224->24224/tcp, 0.0.0.0:24224->24224/udp default_my-app_1 nginx -g daemon off; Up
Test if Kibana web interface is accessible on port 5601 to view logs.
Terminal
curl http://localhost:5601
Expected OutputExpected
<!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <title>Kibana</title> </head> <body> <h1>Welcome to Kibana</h1> </body> </html>
Show logs from the example app container to confirm it is running and generating logs.
Terminal
docker logs default_my-app_1
Expected OutputExpected
172.17.0.1 - - [27/Apr/2024:12:00:00 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.68.0"
Key Concept

If you remember nothing else from this pattern, remember: sending container logs to a central collector like Fluentd lets you search and analyze logs easily from one place.

Common Mistakes
Not configuring the container's logging driver to Fluentd.
Without this, logs stay inside the container and are not sent to the central system.
Add the logging section with driver fluentd and correct address in the container's service definition.
Not exposing Fluentd ports or misconfiguring Fluentd input.
Fluentd won't receive logs if ports are closed or config is wrong.
Expose ports 24224 TCP and UDP and use a valid fluentd.conf file to accept logs.
Starting Kibana before Elasticsearch is ready.
Kibana depends on Elasticsearch and will fail if Elasticsearch is not running.
Use depends_on in docker-compose or wait until Elasticsearch is fully up before starting Kibana.
Summary
Use docker-compose to start Elasticsearch, Kibana, Fluentd, and your app container.
Configure your app container to send logs to Fluentd using the logging driver.
Verify all containers are running and access Kibana to view centralized logs.