Centralized logging setup in Docker - Step-by-Step Execution

Process Flow - Centralized logging setup

Start Docker Containers

↓

Containers generate logs

↓

Docker logging driver captures logs

↓

Logs sent to centralized logging system

↓

Central system stores and indexes logs

↓

User queries logs via dashboard or CLI

Logs flow from Docker containers through a logging driver to a central system where they are stored and accessed.

Execution Sample

Docker

docker network create logging-net

# Run Elasticsearch

docker run -d --name elasticsearch --net=logging-net elasticsearch:8

# Run Fluentd

docker run -d --name fluentd --net=logging-net -p 24224:24224 fluent/fluentd

# Run app container with logging driver

docker run -d --log-driver=fluentd --log-opt fluentd-address=fluentd:24224 myapp

Sets up a network, runs Elasticsearch and Fluentd containers, then runs an app container sending logs to Fluentd.

Process Table

Step	Action	Component	Result	Notes
1	Create Docker network	docker network create logging-net	Network 'logging-net' created	Allows containers to communicate
2	Start Elasticsearch container	docker run elasticsearch	Elasticsearch running on logging-net	Stores and indexes logs
3	Start Fluentd container	docker run fluentd	Fluentd running on logging-net, listening on port 24224	Receives logs from containers
4	Start app container with Fluentd logging driver	docker run myapp with --log-driver=fluentd	App container running, logs sent to Fluentd	Logs forwarded to Fluentd
5	App generates logs	App container	Logs captured by Docker logging driver	Logs sent to Fluentd
6	Fluentd receives logs	Fluentd container	Logs forwarded to Elasticsearch	Logs indexed and stored
7	User queries logs	User via dashboard or CLI	Logs retrieved from Elasticsearch	User sees centralized logs
8	Exit	N/A	Setup complete and running	Logging flow established

💡 All containers running and logs flowing to centralized system

Status Tracker

Component	Initial State	After Setup	Final State
Docker Network	None	logging-net created	logging-net active
Elasticsearch	Not running	Running on logging-net	Running and indexing logs
Fluentd	Not running	Running on logging-net, port 24224 open	Receiving and forwarding logs
App Container	Not running	Running with Fluentd logging driver	Sending logs to Fluentd

Key Moments - 3 Insights

Why do we create a Docker network for logging?

The network allows containers like Elasticsearch, Fluentd, and the app to communicate securely and directly, as shown in step 1 and used in steps 2-4.

How does the app container send logs to Fluentd?

By specifying the Fluentd logging driver and address in the docker run command (step 4), Docker forwards container logs to Fluentd automatically.

What happens if Fluentd is not running when the app starts?

Logs cannot be forwarded and may be lost or buffered; Fluentd must be running to receive logs as shown in steps 3 and 5.

Visual Quiz - 3 Questions

Look at the execution table, at which step does Fluentd start receiving logs?

A Step 4

B Step 5

C Step 6

D Step 7

💡 Hint

Check the 'Component' and 'Result' columns for Fluentd receiving logs

According to the variable tracker, what is the state of the Elasticsearch container after setup?

A Not running

B Running on logging-net

C Running but not indexing logs

D Stopped

💡 Hint

Look at the 'After Setup' column for Elasticsearch in variable_tracker

If the app container did not specify the Fluentd logging driver, what would change in the execution table?

A Step 5 would show logs not forwarded to Fluentd

B Step 5 would show logs not captured by Docker logging driver

C Step 5 would show logs sent to Fluentd

D Step 6 would show Fluentd receiving logs

💡 Hint

Refer to step 5 and 6 about log forwarding from app to Fluentd

Concept Snapshot

Centralized logging in Docker:
- Create a Docker network for container communication
- Run logging system containers (e.g., Elasticsearch, Fluentd) on the network
- Run app containers with logging driver set to Fluentd
- Logs flow from app -> Fluentd -> Elasticsearch
- Query logs centrally via dashboard or CLI
- Ensures easy log management and troubleshooting

Full Transcript

This visual execution shows how to set up centralized logging with Docker. First, a Docker network named 'logging-net' is created to allow containers to communicate. Then, Elasticsearch and Fluentd containers are started on this network. The app container is run with the Fluentd logging driver configured to send logs to Fluentd. As the app runs, it generates logs that Docker captures and forwards to Fluentd. Fluentd receives these logs and sends them to Elasticsearch for storage and indexing. Finally, users can query logs from Elasticsearch using dashboards or command line tools. The execution table traces each step, showing the state changes and flow of logs. The variable tracker summarizes the status of each component before and after setup. Key moments clarify common confusions about network creation, log forwarding, and Fluentd availability. The quiz tests understanding of when logs are received, container states, and effects of missing logging driver configuration.

Practice

1. What is the main purpose of using a centralized logging setup in Docker?

easy

A. To reduce the size of container images

B. To increase container startup speed

C. To collect logs from multiple containers in one place for easier monitoring

D. To automatically restart containers on failure

Solution

1. Step 1: Understand centralized logging concept

   Centralized logging means gathering logs from many containers into a single location.

2. Step 2: Identify the benefit

   This helps in easier monitoring and troubleshooting by having all logs accessible in one place.

3. Final Answer:

   To collect logs from multiple containers in one place for easier monitoring -> Option C

4. Quick Check:

   Centralized logging = collect logs centrally [OK]

Hint: Centralized logging means one place for all container logs [OK]

Common Mistakes:

• Confusing logging with container performance
• Thinking it reduces image size
• Assuming it restarts containers automatically

2. Which Docker command option correctly sets the log driver to Fluentd for a container?

easy

A. docker run --log-driver fluentd myimage

B. docker run --log-driver=fluentd myimage

C. docker run --log-driver:fluentd myimage

D. docker run --logdriver=fluentd myimage

Solution

1. Step 1: Recall Docker log driver syntax

   The correct syntax uses double dashes and equals sign: --log-driver=drivername

2. Step 2: Match the option

   docker run --log-driver=fluentd myimage uses --log-driver=fluentd which is correct syntax for setting Fluentd as log driver.

3. Final Answer:

   docker run --log-driver=fluentd myimage -> Option B

4. Quick Check:

   Correct syntax uses '=' after --log-driver [OK]

Hint: Use --log-driver=drivername with equals sign [OK]

Common Mistakes:

• Using space instead of '=' after --log-driver
• Using colon ':' instead of '='
• Misspelling option name

3. Given the command:

docker run --log-driver=fluentd --log-opt fluentd-address=localhost:24224 myapp

What will happen to the container logs?

medium

A. Logs will be sent to a Fluentd server running on localhost port 24224

B. Logs will be stored only inside the container

C. Logs will be sent to Docker's default json-file driver

D. Container will fail to start due to invalid log options

Solution

1. Step 1: Analyze log driver and options

   The command sets log driver to fluentd and specifies fluentd-address as localhost:24224.

2. Step 2: Understand effect on logs

   Logs will be forwarded to the Fluentd server at the given address instead of default storage.

3. Final Answer:

   Logs will be sent to a Fluentd server running on localhost port 24224 -> Option A

4. Quick Check:

   Fluentd driver + address = logs sent to Fluentd [OK]

Hint: Fluentd log driver sends logs to specified Fluentd address [OK]

Common Mistakes:

• Assuming logs stay inside container
• Thinking default json-file driver is used
• Believing the command causes failure

4. You tried to run:

docker run --log-driver=fluentd --log-opt fluentd-address=localhost:24224 myapp

but logs are not appearing in Fluentd. What is a likely cause?

medium

A. The container image 'myapp' does not support logging

B. The --log-driver option must be set after the image name

C. Docker does not support Fluentd as a log driver

D. Fluentd server is not running or unreachable at localhost:24224

Solution

1. Step 1: Check Fluentd server availability

   If Fluentd server is not running or unreachable at the specified address, logs won't be received.

2. Step 2: Verify Docker and options correctness

   Docker supports Fluentd log driver and the command syntax is correct, so these are unlikely causes.

3. Final Answer:

   Fluentd server is not running or unreachable at localhost:24224 -> Option D

4. Quick Check:

   Fluentd server must be running to receive logs [OK]

Hint: Check Fluentd server is running and reachable [OK]

Common Mistakes:

• Blaming container image for logging issues
• Thinking Docker lacks Fluentd support
• Misplacing --log-driver option

5. You want to centralize logs from multiple Docker containers using syslog. Which is the correct way to run a container to send logs to a syslog server at 192.168.1.100 port 514?

hard

A. docker run --log-driver=syslog --log-opt syslog-address=udp://192.168.1.100:514 myapp

B. docker run --log-driver=json-file --log-opt syslog-address=tcp://192.168.1.100:514 myapp

C. docker run --log-driver=fluentd --log-opt syslog-address=udp://192.168.1.100:514 myapp

D. docker run --log-driver=syslog --log-opt syslog-address=tcp://192.168.1.100:514 myapp

Solution

1. Step 1: Identify correct log driver for syslog

   Use --log-driver=syslog to send logs to a syslog server.

2. Step 2: Choose correct syslog protocol and address

   Syslog commonly uses UDP protocol, so syslog-address=udp://192.168.1.100:514 is correct.

3. Step 3: Verify options match

   docker run --log-driver=syslog --log-opt syslog-address=udp://192.168.1.100:514 myapp correctly sets syslog driver and UDP address; others use wrong driver or protocol.

4. Final Answer:

   docker run --log-driver=syslog --log-opt syslog-address=udp://192.168.1.100:514 myapp -> Option A

5. Quick Check:

   Syslog driver + UDP address = correct syslog logging [OK]

Hint: Syslog driver with udp:// address sends logs correctly [OK]

Common Mistakes:

• Using TCP instead of UDP for syslog
• Mixing Fluentd driver with syslog options
• Using json-file driver for centralized logging