Manager and worker nodes in Docker - Deep Dive

Overview - Manager and worker nodes

What is it?

Manager and worker nodes are parts of a Docker Swarm cluster. A manager node controls and manages the cluster, making decisions like scheduling tasks. Worker nodes run the actual containers and follow instructions from the manager. Together, they help run applications across multiple machines smoothly.

Why it matters

Without manager and worker nodes, running many containers on different machines would be chaotic and hard to control. This setup solves the problem of managing container workloads at scale, making applications reliable and easy to update. Without it, developers would struggle to keep apps running smoothly across servers.

Where it fits

Before learning this, you should understand basic Docker concepts like containers and images. After this, you can learn about Docker Swarm services, scaling, and networking in clusters.

Mental Model

Core Idea

Manager nodes organize and control the cluster, while worker nodes do the actual work of running containers.

Think of it like...

Think of a manager node as a team leader who plans and assigns tasks, and worker nodes as team members who carry out those tasks.

Docker Swarm Cluster
┌─────────────┐       ┌─────────────┐
│ Manager Node│──────▶│ Worker Node │
│  (Control)  │       │  (Runs App) │
└─────────────┘       └─────────────┘
       │                    │
       └────────────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Docker Swarm Basics

Concept: Learn what Docker Swarm is and why it uses nodes.

Docker Swarm is a tool to run containers on many machines as one system. It uses nodes, which are machines or servers, to share the work. Nodes can be managers or workers.

Result

You know that a Docker Swarm cluster is made of nodes with different roles.

Understanding the basic roles of nodes helps you see how Docker manages many containers easily.

2

Foundation

Difference Between Manager and Worker Nodes

Concept: Learn the distinct roles of manager and worker nodes.

Manager nodes decide what tasks to run and where. Worker nodes only run the tasks given by managers. Managers keep the cluster healthy and consistent.

Result

You can tell which node does what in a Docker Swarm.

Knowing the separation of duties prevents confusion when managing a cluster.

3

Intermediate

How Manager Nodes Control the Cluster

🤔Before reading on: do you think manager nodes run containers directly or only manage tasks? Commit to your answer.

Concept: Managers handle scheduling, cluster state, and orchestration but may also run containers.

Manager nodes keep track of all nodes and tasks. They decide which worker runs which container. Managers use Raft consensus to agree on cluster state. They can also run containers themselves if configured.

Result

You understand that managers are brains of the cluster, coordinating work and maintaining state.

Understanding manager responsibilities helps you troubleshoot cluster issues and plan scaling.

4

Intermediate

Worker Nodes Execute Tasks Reliably

🤔Before reading on: do you think worker nodes can reject tasks or must always run them? Commit to your answer.

Concept: Workers receive tasks from managers and run containers as instructed, reporting status back.

Worker nodes listen for instructions from managers. They run containers (tasks) and report success or failure. Workers cannot schedule tasks themselves. If a worker fails, managers reschedule tasks elsewhere.

Result

You see workers as reliable executors that keep the app running.

Knowing worker behavior helps in understanding fault tolerance and load distribution.

5

Intermediate

Joining Nodes to a Swarm Cluster

Concept: Learn how nodes join a cluster as manager or worker.

To add a node, you run a join command with a token. There are separate tokens for managers and workers. This controls who can manage the cluster and who just runs tasks.

Result

You can add new machines to your cluster with correct roles.

Understanding join tokens is key to securing and scaling your cluster.

6

Advanced

Manager Node High Availability Setup

🤔Before reading on: do you think one manager node is enough for production? Commit to your answer.

Concept: Learn how multiple managers keep the cluster reliable and consistent.

In production, you run multiple manager nodes to avoid a single point of failure. Managers use Raft consensus to agree on cluster state. If one manager fails, others continue managing without downtime.

Result

You know how to set up a resilient cluster with multiple managers.

Understanding consensus and high availability prevents cluster outages.

7

Expert

Internal Raft Consensus Mechanism

🤔Before reading on: do you think managers communicate directly or use a special protocol to agree? Commit to your answer.

Concept: Explore how managers use Raft protocol to keep cluster state consistent.

Managers form a Raft cluster, exchanging logs of changes. They elect a leader who decides updates. Followers replicate the leader’s log. This ensures all managers have the same view of tasks and nodes.

Result

You understand the fault-tolerant mechanism behind manager coordination.

Knowing Raft explains why cluster state is reliable even if some managers fail.

Under the Hood

Manager nodes run a Raft consensus algorithm to maintain a consistent cluster state. They store the desired state of services and nodes in a distributed log. Worker nodes receive tasks from managers via secure communication and report status. Managers schedule tasks based on resource availability and cluster health.

Why designed this way?

This design separates control and execution for scalability and reliability. Raft consensus ensures managers agree on cluster state, preventing conflicts. Using tokens for joining nodes secures the cluster. Alternatives like single-node control lack fault tolerance.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Manager Node 1│◀─────▶│ Manager Node 2│◀─────▶│ Manager Node 3│
│  (Raft Leader)│       │  (Raft Follower)│     │  (Raft Follower)│
└───────┬───────┘       └───────┬───────┘       └───────┬───────┘
        │                       │                       │
        ▼                       ▼                       ▼
  ┌─────────────┐         ┌─────────────┐         ┌─────────────┐
  │ Worker Node │         │ Worker Node │         │ Worker Node │
  │   (Runs     │         │   (Runs     │         │   (Runs     │
  │ Containers) │         │ Containers) │         │ Containers) │
  └─────────────┘         └─────────────┘         └─────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do you think worker nodes can decide which containers to run on their own? Commit yes or no.

Common Belief: Worker nodes can schedule containers themselves if they have enough resources.

Reality: Only manager nodes decide where containers run; workers just execute assigned tasks.

Why it matters: Believing workers schedule tasks can cause confusion in debugging why containers don’t start as expected.

Quick: Is it safe to run a Docker Swarm with only one manager node in production? Commit yes or no.

Common Belief: One manager node is enough for managing the cluster safely.

Reality: A single manager is a single point of failure; multiple managers are needed for high availability.

Why it matters: Relying on one manager risks cluster downtime if that node fails.

Quick: Do you think manager nodes never run containers? Commit yes or no.

Common Belief: Manager nodes only manage and never run containers.

Reality: Managers can run containers unless configured not to, especially in small clusters.

Why it matters: Assuming managers never run containers can lead to resource conflicts or unexpected behavior.

Quick: Do you think all nodes in a swarm have equal control? Commit yes or no.

Common Belief: All nodes have equal control and responsibilities in the swarm.

Reality: Only manager nodes control the cluster; workers only execute tasks.

Why it matters: Misunderstanding roles can cause security risks and misconfiguration.

Expert Zone

1

Manager nodes use a quorum system; losing majority managers causes cluster to become read-only until restored.

2

Worker nodes can be promoted to managers dynamically, but this affects cluster stability and should be done carefully.

3

Network latency between managers affects Raft consensus speed and cluster responsiveness.

When NOT to use

For very small or single-host setups, Docker Swarm may be overkill; simpler Docker Compose or Kubernetes might be better depending on scale and complexity.

Production Patterns

In production, clusters often have an odd number of managers (3 or 5) for quorum. Workers are scaled based on workload. Managers are isolated from heavy workloads to maintain responsiveness.

Connections

Distributed Consensus Algorithms

Manager nodes use Raft, a distributed consensus algorithm, to agree on cluster state.

Understanding Raft helps grasp how distributed systems maintain consistency despite failures.

Project Management

Manager and worker nodes mirror roles of project managers and team members in organizing and executing tasks.

Seeing cluster roles as human team roles clarifies responsibilities and communication flow.

Supply Chain Logistics

Managers act like logistics planners deciding where goods go; workers are delivery trucks executing the plan.

This connection shows how coordination and execution separate in complex systems.

Common Pitfalls

#1 Trying to run a swarm with only one manager node in production.

Wrong approach:

docker swarm init
# Only one manager node created

Correct approach:

docker swarm init
# Then add at least two more manager nodes with join tokens

Root cause:

Not understanding the need for high availability and quorum in manager nodes.

#2 Using the worker join token to add a manager node.

Wrong approach:

docker swarm join --token SWMTKN-1-worker ...

Correct approach:

docker swarm join --token SWMTKN-1-manager ...

Root cause:

Confusing join tokens for managers and workers leads to wrong node roles.

#3 Assuming worker nodes can schedule their own containers.

Wrong approach:

Manually starting containers on worker nodes without manager scheduling.

Correct approach:

Use 'docker service create' on manager to schedule containers cluster-wide.

Root cause:

Misunderstanding the control flow between manager and worker nodes.

Key Takeaways

Docker Swarm clusters have manager nodes that control and schedule tasks, and worker nodes that run containers.

Managers use a consensus protocol to keep cluster state consistent and reliable.

Worker nodes cannot schedule tasks; they only execute what managers assign.

High availability requires multiple manager nodes to avoid single points of failure.

Properly joining nodes with correct tokens ensures secure and organized cluster growth.

Practice

1. What is the main role of a manager node in a Docker swarm?

easy

A. To monitor network traffic between containers

B. To run the containers and execute tasks

C. To control the swarm and assign tasks to worker nodes

D. To store container images locally

Solution

1. Step 1: Understand the role of manager nodes

   Manager nodes control the swarm and decide which tasks run where.

2. Step 2: Compare with worker nodes

   Worker nodes only run tasks assigned by managers, they don't control the swarm.

3. Final Answer:

   To control the swarm and assign tasks to worker nodes -> Option C

4. Quick Check:

   Manager node role = control and assign tasks [OK]

Hint: Managers control, workers execute tasks [OK]

Common Mistakes:

• Confusing worker nodes as controllers
• Thinking managers run containers directly
• Assuming managers store images only
• Mixing network monitoring with node roles

2. Which of the following commands correctly initializes a Docker swarm as a manager node?

easy

A. docker node create manager

B. docker swarm init

C. docker swarm start manager

D. docker node init

Solution

1. Step 1: Recall the command to start a swarm

   The command docker swarm init initializes the current node as a manager.

2. Step 2: Check other options for correctness

   Other commands like docker node create or docker swarm start do not exist or are invalid.

3. Final Answer:

   docker swarm init -> Option B

4. Quick Check:

   Initialize swarm = docker swarm init [OK]

Hint: Use 'docker swarm init' to start manager node [OK]

Common Mistakes:

• Using non-existent commands
• Confusing node creation with swarm init
• Trying 'docker node init' which is invalid
• Assuming 'docker swarm start' works

3. Given the following commands run on a new machine:

docker swarm init
docker node ls

What will be the output of docker node ls?

medium

A. A list showing one manager node with status 'Ready' and availability 'Active'

B. An error saying no nodes found

C. A list showing one worker node with status 'Down'

D. Empty output with no nodes listed

Solution

1. Step 1: Understand 'docker swarm init' effect

   This command makes the current machine a manager node in the swarm.

2. Step 2: Check 'docker node ls' output

   It lists all nodes in the swarm. Since only one node exists (the manager), it shows that node as 'Ready' and 'Active'.

3. Final Answer:

   A list showing one manager node with status 'Ready' and availability 'Active' -> Option A

4. Quick Check:

   One manager node listed after init = A list showing one manager node with status 'Ready' and availability 'Active' [OK]

Hint: After init, node list shows one active manager [OK]

Common Mistakes:

• Expecting worker nodes without joining
• Thinking output is empty or error
• Confusing node status terms
• Assuming node is down immediately

4. You ran docker swarm join --token SWMTKN-1-xyz 192.168.1.10:2377 on a node but it failed with "Error response from daemon: This node is already part of a swarm." What is the likely cause?

medium

A. The node is already a member of another swarm

B. The token is invalid or expired

C. The manager IP address is incorrect

D. Docker daemon is not running on the node

Solution

1. Step 1: Analyze the error message

   The error says the node is already part of a swarm, so it cannot join another without leaving first.

2. Step 2: Check other options

   Invalid token or wrong IP would cause different errors. Docker daemon not running would prevent any command execution.

3. Final Answer:

   The node is already a member of another swarm -> Option A

4. Quick Check:

   Node already in swarm = join fails with this error [OK]

Hint: Node must leave swarm before joining another [OK]

Common Mistakes:

• Assuming token or IP is always the cause
• Ignoring existing swarm membership
• Not checking Docker daemon status
• Trying to join without leaving first

5. You want to add a new worker node to an existing swarm. The manager node IP is 10.0.0.5 and the join token is SWMTKN-1-abc123. Which sequence of commands correctly adds the worker node and verifies it joined?

hard

A. On worker: docker node add --token SWMTKN-1-abc123 10.0.0.5On manager: docker swarm ls

B. On manager: docker swarm join --token SWMTKN-1-abc123 10.0.0.5:2377On worker: docker node ls

C. On manager: docker node join --token SWMTKN-1-abc123 10.0.0.5On worker: docker swarm ls

D. On worker: docker swarm join --token SWMTKN-1-abc123 10.0.0.5:2377On manager: docker node ls

Solution

1. Step 1: Join the worker node to the swarm

   The worker node uses docker swarm join --token TOKEN MANAGER_IP:2377 to join the swarm.

2. Step 2: Verify the node joined on the manager

   On the manager node, docker node ls lists all nodes including the new worker.

3. Final Answer:

   On worker: docker swarm join --token SWMTKN-1-abc123 10.0.0.5:2377; On manager: docker node ls -> Option D

4. Quick Check:

   Join on worker, list on manager = correct process [OK]

Hint: Join on worker, verify on manager with 'docker node ls' [OK]

Common Mistakes:

• Running join on manager instead of worker
• Using wrong commands like 'docker node add'
• Trying to list nodes on worker node
• Omitting port 2377 in join command