Standalone vs distributed mode in Kafka - Visual Side-by-Side Comparison

Process Flow - Standalone vs distributed mode

Start Kafka

↓

Choose Mode

↓

Standalone

↓

Single Broker

↓

Simple Setup

↓

Local Use

↓

Limited

Kafka can run in two modes: standalone with one broker for simple local use, or distributed with multiple brokers for scalability and fault tolerance.

Execution Sample

Kafka

1. Start Kafka in standalone mode
2. Start Kafka in distributed mode
3. Send message to topic
4. Consume message from topic

This shows starting Kafka in two modes and sending/receiving messages to illustrate differences.

Process Table

Step	Action	Mode	Kafka Brokers	Result
1	Start Kafka	Standalone	1 broker	Kafka runs locally with one broker
2	Send message to topic	Standalone	1 broker	Message stored in single broker
3	Consume message	Standalone	1 broker	Message consumed from single broker
4	Start Kafka	Distributed	3 brokers	Kafka cluster runs with multiple brokers
5	Send message to topic	Distributed	3 brokers	Message partitioned and replicated across brokers
6	Consume message	Distributed	3 brokers	Message consumed from cluster with fault tolerance
7	Broker failure	Distributed	2 brokers active	Cluster continues working without data loss
8	Broker failure	Standalone	0 brokers active	Kafka stops, no fault tolerance

💡 Execution stops after showing message flow and fault tolerance differences in both modes.

Status Tracker

Variable	Start	After Step 2	After Step 5	After Step 7	Final
Kafka Brokers	0	1 (Standalone)	3 (Distributed)	2 (Distributed after failure)	Depends on mode
Message Stored	None	Stored in 1 broker	Partitioned & replicated	Still available	Consumed successfully
Fault Tolerance	None	None	Enabled	Cluster active	Depends on mode

Key Moments - 3 Insights

Why does Kafka stop working in standalone mode if the broker fails?

Because in standalone mode there is only one broker (see execution_table step 8), if it fails, no other broker can take over.

How does distributed mode keep messages safe when a broker fails?

Distributed mode replicates messages across multiple brokers (execution_table step 5), so if one fails (step 7), others still have the data.

Is standalone mode suitable for production with many users?

No, standalone mode is simple and local (execution_table steps 1-3), but it lacks scalability and fault tolerance needed for production.

Visual Quiz - 3 Questions

Look at the execution_table, how many brokers are active after a failure in distributed mode?

A 1 broker

B 0 brokers

C 2 brokers

D 3 brokers

💡 Hint

Check execution_table row 7 under 'Kafka Brokers' column.

At which step does Kafka start running as a cluster with multiple brokers?

A Step 1

B Step 4

C Step 2

D Step 6

💡 Hint

Look at execution_table row 4 for starting Kafka in distributed mode.

If you want fault tolerance, which mode should you choose according to the variable_tracker?

A Distributed mode

B Either mode

C Standalone mode

D Neither mode

💡 Hint

See 'Fault Tolerance' row in variable_tracker after Step 5 and Step 7.

Concept Snapshot

Kafka Modes:
- Standalone: 1 broker, simple, local use
- Distributed: multiple brokers, scalable, fault tolerant
- Standalone stops if broker fails
- Distributed replicates data for safety
- Use distributed for production and high availability

Full Transcript

Kafka can run in standalone mode with a single broker or in distributed mode with multiple brokers. Standalone mode is simple and good for local testing but has no fault tolerance. Distributed mode uses multiple brokers to partition and replicate data, providing scalability and fault tolerance. If a broker fails in distributed mode, the cluster continues working without data loss. In standalone mode, failure stops Kafka. This visual trace shows starting Kafka in both modes, sending and consuming messages, and how fault tolerance works in distributed mode.

Practice

1. What is the main purpose of running Kafka Connect in standalone mode?

easy

A. To run multiple workers for high scalability

B. To run Kafka Connect in a single process for simple or testing use

C. To enable fault tolerance across multiple machines

D. To connect Kafka with distributed databases automatically

Solution

1. Step 1: Understand standalone mode context

   Standalone mode runs Kafka Connect in one process, suitable for simple or testing scenarios.

2. Step 2: Compare with distributed mode

   Distributed mode runs multiple workers for scalability and fault tolerance, unlike standalone mode.

3. Final Answer:

   To run Kafka Connect in a single process for simple or testing use -> Option B

4. Quick Check:

   Standalone mode = single process for testing [OK]

Hint: Standalone = single process, simple use [OK]

Common Mistakes:

• Confusing standalone with distributed mode
• Thinking standalone supports fault tolerance
• Assuming standalone is for large scale pipelines

2. Which of the following is the correct command to start Kafka Connect in distributed mode?

easy

A. kafka-connect.sh start-standalone

B. connect-standalone.sh config/connect-standalone.properties

C. kafka-connect.sh start-distributed

D. connect-distributed.sh config/connect-distributed.properties

Solution

1. Step 1: Identify the script for distributed mode

   The script connect-distributed.sh is used to start Kafka Connect in distributed mode.

2. Step 2: Confirm the configuration file

   The configuration file config/connect-distributed.properties is the standard config for distributed mode.

3. Final Answer:

   connect-distributed.sh config/connect-distributed.properties -> Option D

4. Quick Check:

   Distributed mode uses connect-distributed.sh [OK]

Hint: Distributed mode uses connect-distributed.sh script [OK]

Common Mistakes:

• Using connect-standalone.sh for distributed mode
• Mixing config files between modes
• Assuming kafka-connect.sh is a valid command

3. Given the following scenario: You start Kafka Connect with connect-standalone.sh and add two connectors. What will happen if the process crashes?

medium

A. Both connectors will automatically restart on another worker

B. Connectors will continue running without interruption

C. The connectors will stop and require manual restart

D. Kafka Connect will switch to distributed mode automatically

Solution

1. Step 1: Understand standalone mode fault tolerance

   Standalone mode runs in a single process without fault tolerance or automatic recovery.

2. Step 2: Consequence of process crash

   If the process crashes, connectors stop and need manual restart because no other worker is available.

3. Final Answer:

   The connectors will stop and require manual restart -> Option C

4. Quick Check:

   Standalone mode lacks automatic recovery [OK]

Hint: Standalone mode needs manual restart after crash [OK]

Common Mistakes:

• Assuming automatic failover in standalone mode
• Thinking distributed mode starts automatically
• Believing connectors run independently of the process

4. You configured Kafka Connect in distributed mode but notice that only one worker is running and no failover happens. What is the likely cause?

medium

A. You started Kafka Connect with connect-standalone.sh instead of connect-distributed.sh

B. You did not configure the Kafka cluster correctly

C. You forgot to add connectors to the worker

D. Distributed mode does not support multiple workers

Solution

1. Step 1: Check the startup script used

   If connect-standalone.sh is used, only one worker runs without failover, even if config is for distributed mode.

2. Step 2: Understand distributed mode requirements

   Distributed mode requires starting with connect-distributed.sh to run multiple workers and enable failover.

3. Final Answer:

   You started Kafka Connect with connect-standalone.sh instead of connect-distributed.sh -> Option A

4. Quick Check:

   Wrong startup script causes single worker [OK]

Hint: Use connect-distributed.sh to run multiple workers [OK]

Common Mistakes:

• Assuming config alone enables distributed mode
• Believing distributed mode runs with standalone script
• Ignoring startup script importance

5. You have a data pipeline that must handle high volume and remain available even if one worker fails. Which Kafka Connect mode should you choose and why?

hard

A. Distributed mode, because it supports multiple workers and fault tolerance

B. Standalone mode, because it is simpler to configure

C. Standalone mode, because it automatically restarts connectors on failure

D. Distributed mode, because it runs connectors in a single process

Solution

1. Step 1: Analyze pipeline requirements

   High volume and availability require scalability and fault tolerance.

2. Step 2: Match requirements to Kafka Connect modes

   Distributed mode runs multiple workers and supports fault tolerance, meeting these needs.

3. Final Answer:

   Distributed mode, because it supports multiple workers and fault tolerance -> Option A

4. Quick Check:

   High volume + fault tolerance = distributed mode [OK]

Hint: Choose distributed mode for scalability and fault tolerance [OK]

Common Mistakes:

• Choosing standalone for high availability
• Thinking standalone auto-restarts connectors
• Misunderstanding distributed mode runs single process