Centralized logging setup in Docker - Commands & Configuration

Introduction

When you run many containers, it is hard to find logs from each one separately. Centralized logging collects all logs in one place so you can see and search them easily.

When you want to see logs from multiple containers in one dashboard.

When you need to keep logs safe even if containers stop or restart.

When you want to analyze logs to find errors or performance issues.

When you want to share logs with your team without accessing each server.

When you want to keep logs for a long time without filling container storage.

Config File - docker-compose.yml

docker-compose.yml

version: '3.8'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.6.3
    environment:
      - discovery.type=single-node
    ports:
      - "9200:9200"
  kibana:
    image: docker.elastic.co/kibana/kibana:8.6.3
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch
  fluentd:
    image: fluent/fluentd:v1.15-debian-1
    volumes:
      - ./fluentd.conf:/fluentd/etc/fluentd.conf
    ports:
      - "24224:24224"
      - "24224:24224/udp"
  my-app:
    image: nginx:1.23
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        tag: my-app

This file sets up four services:

• Elasticsearch: stores logs and allows searching.
• Kibana: web interface to view and analyze logs.
• Fluentd: collects logs from containers and sends to Elasticsearch.
• my-app: example app container sending logs to Fluentd.

The logging section in my-app tells Docker to send logs to Fluentd on port 24224.

Commands

Start all services in the background: Elasticsearch, Kibana, Fluentd, and the example app.

Terminal

docker-compose up -d

Expected Output

Creating network "default" with the default driver

Creating volume "default_elasticsearch_data" with default driver

Creating default_elasticsearch_1 ... done

Creating default_kibana_1       ... done

Creating default_fluentd_1      ... done

Creating default_my-app_1       ... done

→

-d - Run containers in detached mode (in background)

Check that all containers are running properly after starting them.

Terminal

docker-compose ps

Expected Output

    Name                   Command               State           Ports         

default_elasticsearch_1   /bin/tini -- /usr/local/bin/docker-entrypoint.sh elasticsearch   Up      0.0.0.0:9200->9200/tcp



default_kibana_1          /bin/tini -- /usr/local/bin/kibana-docker   Up      0.0.0.0:5601->5601/tcp



default_fluentd_1         fluentd -c /fluentd/etc/fluentd.conf      Up      0.0.0.0:24224->24224/tcp, 0.0.0.0:24224->24224/udp



default_my-app_1          nginx -g daemon off;                      Up      

Test if Kibana web interface is accessible on port 5601 to view logs.

Terminal

curl http://localhost:5601

Expected Output

<!DOCTYPE html>

<html lang="en">

<head>

  <meta charset="utf-8">

  <title>Kibana</title>

</head>

<body>

  <h1>Welcome to Kibana</h1>

</body>

</html>

Show logs from the example app container to confirm it is running and generating logs.

Terminal

docker logs default_my-app_1

Expected Output

172.17.0.1 - - [27/Apr/2024:12:00:00 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.68.0"

Key Concept

If you remember nothing else from this pattern, remember: sending container logs to a central collector like Fluentd lets you search and analyze logs easily from one place.

Common Mistakes

Not configuring the container's logging driver to Fluentd.

Without this, logs stay inside the container and are not sent to the central system.

Add the logging section with driver fluentd and correct address in the container's service definition.

Not exposing Fluentd ports or misconfiguring Fluentd input.

Fluentd won't receive logs if ports are closed or config is wrong.

Expose ports 24224 TCP and UDP and use a valid fluentd.conf file to accept logs.

Starting Kibana before Elasticsearch is ready.

Kibana depends on Elasticsearch and will fail if Elasticsearch is not running.

Use depends_on in docker-compose or wait until Elasticsearch is fully up before starting Kibana.

Summary

Use docker-compose to start Elasticsearch, Kibana, Fluentd, and your app container.

Configure your app container to send logs to Fluentd using the logging driver.

Verify all containers are running and access Kibana to view centralized logs.

Practice

1. What is the main purpose of using a centralized logging setup in Docker?

easy

A. To reduce the size of container images

B. To increase container startup speed

C. To collect logs from multiple containers in one place for easier monitoring

D. To automatically restart containers on failure

Solution

1. Step 1: Understand centralized logging concept

   Centralized logging means gathering logs from many containers into a single location.

2. Step 2: Identify the benefit

   This helps in easier monitoring and troubleshooting by having all logs accessible in one place.

3. Final Answer:

   To collect logs from multiple containers in one place for easier monitoring -> Option C

4. Quick Check:

   Centralized logging = collect logs centrally [OK]

Hint: Centralized logging means one place for all container logs [OK]

Common Mistakes:

• Confusing logging with container performance
• Thinking it reduces image size
• Assuming it restarts containers automatically

2. Which Docker command option correctly sets the log driver to Fluentd for a container?

easy

A. docker run --log-driver fluentd myimage

B. docker run --log-driver=fluentd myimage

C. docker run --log-driver:fluentd myimage

D. docker run --logdriver=fluentd myimage

Solution

1. Step 1: Recall Docker log driver syntax

   The correct syntax uses double dashes and equals sign: --log-driver=drivername

2. Step 2: Match the option

   docker run --log-driver=fluentd myimage uses --log-driver=fluentd which is correct syntax for setting Fluentd as log driver.

3. Final Answer:

   docker run --log-driver=fluentd myimage -> Option B

4. Quick Check:

   Correct syntax uses '=' after --log-driver [OK]

Hint: Use --log-driver=drivername with equals sign [OK]

Common Mistakes:

• Using space instead of '=' after --log-driver
• Using colon ':' instead of '='
• Misspelling option name

3. Given the command:

docker run --log-driver=fluentd --log-opt fluentd-address=localhost:24224 myapp

What will happen to the container logs?

medium

A. Logs will be sent to a Fluentd server running on localhost port 24224

B. Logs will be stored only inside the container

C. Logs will be sent to Docker's default json-file driver

D. Container will fail to start due to invalid log options

Solution

1. Step 1: Analyze log driver and options

   The command sets log driver to fluentd and specifies fluentd-address as localhost:24224.

2. Step 2: Understand effect on logs

   Logs will be forwarded to the Fluentd server at the given address instead of default storage.

3. Final Answer:

   Logs will be sent to a Fluentd server running on localhost port 24224 -> Option A

4. Quick Check:

   Fluentd driver + address = logs sent to Fluentd [OK]

Hint: Fluentd log driver sends logs to specified Fluentd address [OK]

Common Mistakes:

• Assuming logs stay inside container
• Thinking default json-file driver is used
• Believing the command causes failure

4. You tried to run:

docker run --log-driver=fluentd --log-opt fluentd-address=localhost:24224 myapp

but logs are not appearing in Fluentd. What is a likely cause?

medium

A. The container image 'myapp' does not support logging

B. The --log-driver option must be set after the image name

C. Docker does not support Fluentd as a log driver

D. Fluentd server is not running or unreachable at localhost:24224

Solution

1. Step 1: Check Fluentd server availability

   If Fluentd server is not running or unreachable at the specified address, logs won't be received.

2. Step 2: Verify Docker and options correctness

   Docker supports Fluentd log driver and the command syntax is correct, so these are unlikely causes.

3. Final Answer:

   Fluentd server is not running or unreachable at localhost:24224 -> Option D

4. Quick Check:

   Fluentd server must be running to receive logs [OK]

Hint: Check Fluentd server is running and reachable [OK]

Common Mistakes:

• Blaming container image for logging issues
• Thinking Docker lacks Fluentd support
• Misplacing --log-driver option

5. You want to centralize logs from multiple Docker containers using syslog. Which is the correct way to run a container to send logs to a syslog server at 192.168.1.100 port 514?

hard

A. docker run --log-driver=syslog --log-opt syslog-address=udp://192.168.1.100:514 myapp

B. docker run --log-driver=json-file --log-opt syslog-address=tcp://192.168.1.100:514 myapp

C. docker run --log-driver=fluentd --log-opt syslog-address=udp://192.168.1.100:514 myapp

D. docker run --log-driver=syslog --log-opt syslog-address=tcp://192.168.1.100:514 myapp

Solution

1. Step 1: Identify correct log driver for syslog

   Use --log-driver=syslog to send logs to a syslog server.

2. Step 2: Choose correct syslog protocol and address

   Syslog commonly uses UDP protocol, so syslog-address=udp://192.168.1.100:514 is correct.

3. Step 3: Verify options match

   docker run --log-driver=syslog --log-opt syslog-address=udp://192.168.1.100:514 myapp correctly sets syslog driver and UDP address; others use wrong driver or protocol.

4. Final Answer:

   docker run --log-driver=syslog --log-opt syslog-address=udp://192.168.1.100:514 myapp -> Option A

5. Quick Check:

   Syslog driver + UDP address = correct syslog logging [OK]

Hint: Syslog driver with udp:// address sends logs correctly [OK]

Common Mistakes:

• Using TCP instead of UDP for syslog
• Mixing Fluentd driver with syslog options
• Using json-file driver for centralized logging