Manager and worker nodes in Docker - Deep Dive

Overview - Manager and worker nodes

What is it?

Manager and worker nodes are parts of a Docker Swarm cluster. A manager node controls and manages the cluster, making decisions like scheduling tasks. Worker nodes run the actual containers and follow instructions from the manager. Together, they help run applications across multiple machines smoothly.

Why it matters

Without manager and worker nodes, running many containers on different machines would be chaotic and hard to control. This setup solves the problem of managing container workloads at scale, making applications reliable and easy to update. Without it, developers would struggle to keep apps running smoothly across servers.

Where it fits

Before learning this, you should understand basic Docker concepts like containers and images. After this, you can learn about Docker Swarm services, scaling, and networking in clusters.

Mental Model

Core Idea

Manager nodes organize and control the cluster, while worker nodes do the actual work of running containers.

Think of it like...

Think of a manager node as a team leader who plans and assigns tasks, and worker nodes as team members who carry out those tasks.

Docker Swarm Cluster
┌─────────────┐       ┌─────────────┐
│ Manager Node│──────▶│ Worker Node │
│  (Control)  │       │  (Runs App) │
└─────────────┘       └─────────────┘
       │                    │
       └────────────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Docker Swarm Basics

Concept: Learn what Docker Swarm is and why it uses nodes.

Docker Swarm is a tool to run containers on many machines as one system. It uses nodes, which are machines or servers, to share the work. Nodes can be managers or workers.

Result

You know that a Docker Swarm cluster is made of nodes with different roles.

Understanding the basic roles of nodes helps you see how Docker manages many containers easily.

2

Foundation

Difference Between Manager and Worker Nodes

Concept: Learn the distinct roles of manager and worker nodes.

Manager nodes decide what tasks to run and where. Worker nodes only run the tasks given by managers. Managers keep the cluster healthy and consistent.

Result

You can tell which node does what in a Docker Swarm.

Knowing the separation of duties prevents confusion when managing a cluster.

3

Intermediate

How Manager Nodes Control the Cluster

🤔Before reading on: do you think manager nodes run containers directly or only manage tasks? Commit to your answer.

Concept: Managers handle scheduling, cluster state, and orchestration but may also run containers.

Manager nodes keep track of all nodes and tasks. They decide which worker runs which container. Managers use Raft consensus to agree on cluster state. They can also run containers themselves if configured.

Result

You understand that managers are brains of the cluster, coordinating work and maintaining state.

Understanding manager responsibilities helps you troubleshoot cluster issues and plan scaling.

4

Intermediate

Worker Nodes Execute Tasks Reliably

🤔Before reading on: do you think worker nodes can reject tasks or must always run them? Commit to your answer.

Concept: Workers receive tasks from managers and run containers as instructed, reporting status back.

Worker nodes listen for instructions from managers. They run containers (tasks) and report success or failure. Workers cannot schedule tasks themselves. If a worker fails, managers reschedule tasks elsewhere.

Result

You see workers as reliable executors that keep the app running.

Knowing worker behavior helps in understanding fault tolerance and load distribution.

5

Intermediate

Joining Nodes to a Swarm Cluster

Concept: Learn how nodes join a cluster as manager or worker.

To add a node, you run a join command with a token. There are separate tokens for managers and workers. This controls who can manage the cluster and who just runs tasks.

Result

You can add new machines to your cluster with correct roles.

Understanding join tokens is key to securing and scaling your cluster.

6

Advanced

Manager Node High Availability Setup

🤔Before reading on: do you think one manager node is enough for production? Commit to your answer.

Concept: Learn how multiple managers keep the cluster reliable and consistent.

In production, you run multiple manager nodes to avoid a single point of failure. Managers use Raft consensus to agree on cluster state. If one manager fails, others continue managing without downtime.

Result

You know how to set up a resilient cluster with multiple managers.

Understanding consensus and high availability prevents cluster outages.

7

Expert

Internal Raft Consensus Mechanism

🤔Before reading on: do you think managers communicate directly or use a special protocol to agree? Commit to your answer.

Concept: Explore how managers use Raft protocol to keep cluster state consistent.

Managers form a Raft cluster, exchanging logs of changes. They elect a leader who decides updates. Followers replicate the leader’s log. This ensures all managers have the same view of tasks and nodes.

Result

You understand the fault-tolerant mechanism behind manager coordination.

Knowing Raft explains why cluster state is reliable even if some managers fail.

Under the Hood

Manager nodes run a Raft consensus algorithm to maintain a consistent cluster state. They store the desired state of services and nodes in a distributed log. Worker nodes receive tasks from managers via secure communication and report status. Managers schedule tasks based on resource availability and cluster health.

Why designed this way?

This design separates control and execution for scalability and reliability. Raft consensus ensures managers agree on cluster state, preventing conflicts. Using tokens for joining nodes secures the cluster. Alternatives like single-node control lack fault tolerance.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Manager Node 1│◀─────▶│ Manager Node 2│◀─────▶│ Manager Node 3│
│  (Raft Leader)│       │  (Raft Follower)│     │  (Raft Follower)│
└───────┬───────┘       └───────┬───────┘       └───────┬───────┘
        │                       │                       │
        ▼                       ▼                       ▼
  ┌─────────────┐         ┌─────────────┐         ┌─────────────┐
  │ Worker Node │         │ Worker Node │         │ Worker Node │
  │   (Runs     │         │   (Runs     │         │   (Runs     │
  │ Containers) │         │ Containers) │         │ Containers) │
  └─────────────┘         └─────────────┘         └─────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do you think worker nodes can decide which containers to run on their own? Commit yes or no.

Common Belief: Worker nodes can schedule containers themselves if they have enough resources.

Reality: Only manager nodes decide where containers run; workers just execute assigned tasks.

Why it matters: Believing workers schedule tasks can cause confusion in debugging why containers don’t start as expected.

Quick: Is it safe to run a Docker Swarm with only one manager node in production? Commit yes or no.

Common Belief: One manager node is enough for managing the cluster safely.

Reality: A single manager is a single point of failure; multiple managers are needed for high availability.

Why it matters: Relying on one manager risks cluster downtime if that node fails.

Quick: Do you think manager nodes never run containers? Commit yes or no.

Common Belief: Manager nodes only manage and never run containers.

Reality: Managers can run containers unless configured not to, especially in small clusters.

Why it matters: Assuming managers never run containers can lead to resource conflicts or unexpected behavior.

Quick: Do you think all nodes in a swarm have equal control? Commit yes or no.

Common Belief: All nodes have equal control and responsibilities in the swarm.

Reality: Only manager nodes control the cluster; workers only execute tasks.

Why it matters: Misunderstanding roles can cause security risks and misconfiguration.

Expert Zone

1

Manager nodes use a quorum system; losing majority managers causes cluster to become read-only until restored.

2

Worker nodes can be promoted to managers dynamically, but this affects cluster stability and should be done carefully.

3

Network latency between managers affects Raft consensus speed and cluster responsiveness.

When NOT to use

For very small or single-host setups, Docker Swarm may be overkill; simpler Docker Compose or Kubernetes might be better depending on scale and complexity.

Production Patterns

In production, clusters often have an odd number of managers (3 or 5) for quorum. Workers are scaled based on workload. Managers are isolated from heavy workloads to maintain responsiveness.

Connections

Distributed Consensus Algorithms

Manager nodes use Raft, a distributed consensus algorithm, to agree on cluster state.

Understanding Raft helps grasp how distributed systems maintain consistency despite failures.

Project Management

Manager and worker nodes mirror roles of project managers and team members in organizing and executing tasks.

Seeing cluster roles as human team roles clarifies responsibilities and communication flow.

Supply Chain Logistics

Managers act like logistics planners deciding where goods go; workers are delivery trucks executing the plan.

This connection shows how coordination and execution separate in complex systems.

Common Pitfalls

#1 Trying to run a swarm with only one manager node in production.

Wrong approach:

docker swarm init
# Only one manager node created

Correct approach:

docker swarm init
# Then add at least two more manager nodes with join tokens

Root cause:

Not understanding the need for high availability and quorum in manager nodes.

#2 Using the worker join token to add a manager node.

Wrong approach:

docker swarm join --token SWMTKN-1-worker ...

Correct approach:

docker swarm join --token SWMTKN-1-manager ...

Root cause:

Confusing join tokens for managers and workers leads to wrong node roles.

#3 Assuming worker nodes can schedule their own containers.

Wrong approach:

Manually starting containers on worker nodes without manager scheduling.

Correct approach:

Use 'docker service create' on manager to schedule containers cluster-wide.

Root cause:

Misunderstanding the control flow between manager and worker nodes.

Key Takeaways

Docker Swarm clusters have manager nodes that control and schedule tasks, and worker nodes that run containers.

Managers use a consensus protocol to keep cluster state consistent and reliable.

Worker nodes cannot schedule tasks; they only execute what managers assign.

High availability requires multiple manager nodes to avoid single points of failure.

Properly joining nodes with correct tokens ensures secure and organized cluster growth.