Centralized logging setup in Docker - Step-by-Step Execution

Process Flow - Centralized logging setup

Start Docker Containers

↓

Containers generate logs

↓

Docker logging driver captures logs

↓

Logs sent to centralized logging system

↓

Central system stores and indexes logs

↓

User queries logs via dashboard or CLI

Logs flow from Docker containers through a logging driver to a central system where they are stored and accessed.

Execution Sample

Docker

docker network create logging-net

# Run Elasticsearch

docker run -d --name elasticsearch --net=logging-net elasticsearch:8

# Run Fluentd

docker run -d --name fluentd --net=logging-net -p 24224:24224 fluent/fluentd

# Run app container with logging driver

docker run -d --log-driver=fluentd --log-opt fluentd-address=fluentd:24224 myapp

Sets up a network, runs Elasticsearch and Fluentd containers, then runs an app container sending logs to Fluentd.

Process Table

Step	Action	Component	Result	Notes
1	Create Docker network	docker network create logging-net	Network 'logging-net' created	Allows containers to communicate
2	Start Elasticsearch container	docker run elasticsearch	Elasticsearch running on logging-net	Stores and indexes logs
3	Start Fluentd container	docker run fluentd	Fluentd running on logging-net, listening on port 24224	Receives logs from containers
4	Start app container with Fluentd logging driver	docker run myapp with --log-driver=fluentd	App container running, logs sent to Fluentd	Logs forwarded to Fluentd
5	App generates logs	App container	Logs captured by Docker logging driver	Logs sent to Fluentd
6	Fluentd receives logs	Fluentd container	Logs forwarded to Elasticsearch	Logs indexed and stored
7	User queries logs	User via dashboard or CLI	Logs retrieved from Elasticsearch	User sees centralized logs
8	Exit	N/A	Setup complete and running	Logging flow established

💡 All containers running and logs flowing to centralized system

Status Tracker

Component	Initial State	After Setup	Final State
Docker Network	None	logging-net created	logging-net active
Elasticsearch	Not running	Running on logging-net	Running and indexing logs
Fluentd	Not running	Running on logging-net, port 24224 open	Receiving and forwarding logs
App Container	Not running	Running with Fluentd logging driver	Sending logs to Fluentd

Key Moments - 3 Insights

Why do we create a Docker network for logging?

The network allows containers like Elasticsearch, Fluentd, and the app to communicate securely and directly, as shown in step 1 and used in steps 2-4.

How does the app container send logs to Fluentd?

By specifying the Fluentd logging driver and address in the docker run command (step 4), Docker forwards container logs to Fluentd automatically.

What happens if Fluentd is not running when the app starts?

Logs cannot be forwarded and may be lost or buffered; Fluentd must be running to receive logs as shown in steps 3 and 5.

Visual Quiz - 3 Questions

Look at the execution table, at which step does Fluentd start receiving logs?

A Step 4

B Step 5

C Step 6

D Step 7

💡 Hint

Check the 'Component' and 'Result' columns for Fluentd receiving logs

According to the variable tracker, what is the state of the Elasticsearch container after setup?

A Not running

B Running on logging-net

C Running but not indexing logs

D Stopped

💡 Hint

Look at the 'After Setup' column for Elasticsearch in variable_tracker

If the app container did not specify the Fluentd logging driver, what would change in the execution table?

A Step 5 would show logs not forwarded to Fluentd

B Step 5 would show logs not captured by Docker logging driver

C Step 5 would show logs sent to Fluentd

D Step 6 would show Fluentd receiving logs

💡 Hint

Refer to step 5 and 6 about log forwarding from app to Fluentd

Concept Snapshot

Centralized logging in Docker:
- Create a Docker network for container communication
- Run logging system containers (e.g., Elasticsearch, Fluentd) on the network
- Run app containers with logging driver set to Fluentd
- Logs flow from app -> Fluentd -> Elasticsearch
- Query logs centrally via dashboard or CLI
- Ensures easy log management and troubleshooting

Full Transcript

This visual execution shows how to set up centralized logging with Docker. First, a Docker network named 'logging-net' is created to allow containers to communicate. Then, Elasticsearch and Fluentd containers are started on this network. The app container is run with the Fluentd logging driver configured to send logs to Fluentd. As the app runs, it generates logs that Docker captures and forwards to Fluentd. Fluentd receives these logs and sends them to Elasticsearch for storage and indexing. Finally, users can query logs from Elasticsearch using dashboards or command line tools. The execution table traces each step, showing the state changes and flow of logs. The variable tracker summarizes the status of each component before and after setup. Key moments clarify common confusions about network creation, log forwarding, and Fluentd availability. The quiz tests understanding of when logs are received, container states, and effects of missing logging driver configuration.