Standalone vs distributed mode in Kafka - CLI Comparison

Introduction

Kafka can run in two ways: standalone mode for simple testing and learning, and distributed mode for real-world use with multiple servers working together. Standalone mode runs everything on one machine, while distributed mode spreads tasks across many machines to handle more data and stay reliable.

When you want to try Kafka quickly on your laptop without setting up multiple servers

When you need to test your Kafka setup or application code before going live

When you want to run Kafka in production with high availability and fault tolerance

When you want to handle large volumes of data across multiple servers

When you want Kafka to keep working even if some servers fail

Commands

Starts Kafka in standalone mode using the default single-node configuration file.

Terminal

kafka-server-start.sh /usr/local/kafka/config/server.properties

Expected Output

[2024-06-01 12:00:00,000] INFO Kafka started (kafka.server.KafkaServer)

[2024-06-01 12:00:00,001] INFO Awaiting socket connections on port 9092 (kafka.network.SocketServer)

Creates a topic named 'test-topic' with one partition and no replication, suitable for standalone mode.

Terminal

kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Expected Output

Created topic test-topic.

→

--partitions - Number of partitions for the topic

→

--replication-factor - Number of copies of data for fault tolerance

Starts three Kafka servers in distributed mode, each with its own configuration file to run on different ports and form a cluster.

Terminal

kafka-server-start.sh /usr/local/kafka/config/server-1.properties &
kafka-server-start.sh /usr/local/kafka/config/server-2.properties &
kafka-server-start.sh /usr/local/kafka/config/server-3.properties &

Expected Output

[2024-06-01 12:01:00,000] INFO Kafka server started on port 9092 (server-1)

[2024-06-01 12:01:00,100] INFO Kafka server started on port 9093 (server-2)

[2024-06-01 12:01:00,200] INFO Kafka server started on port 9094 (server-3)

Creates a topic named 'distributed-topic' with three partitions and replication factor three for fault tolerance in distributed mode.

Terminal

kafka-topics.sh --create --topic distributed-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 3

Expected Output

Created topic distributed-topic.

→

--partitions - Number of partitions to spread load

→

--replication-factor - Number of copies to keep data safe

Shows details about the distributed topic, including partitions and replicas, to verify the distributed setup.

Terminal

kafka-topics.sh --describe --topic distributed-topic --bootstrap-server localhost:9092

Expected Output

Topic: distributed-topic	PartitionCount: 3	ReplicationFactor: 3	Configs: 

	Partition: 0	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3

	Partition: 1	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1

	Partition: 2	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2

Key Concept

If you remember nothing else from this pattern, remember: standalone mode is for simple testing on one machine, while distributed mode uses multiple servers to handle more data and stay reliable.

Common Mistakes

Trying to use replication in standalone mode by setting replication-factor greater than 1

Standalone mode runs only one Kafka server, so replication cannot work and the command will fail.

Use replication-factor 1 in standalone mode or switch to distributed mode with multiple servers for replication.

Starting multiple Kafka servers with the same configuration file and port

Multiple servers cannot run on the same port and configuration, causing startup errors.

Create separate configuration files with different ports and broker IDs for each Kafka server in distributed mode.

Summary

Start Kafka with a single server for quick testing using the default config file.

Create topics with one partition and no replication in standalone mode.

Run multiple Kafka servers with different configs to form a distributed cluster.

Create topics with multiple partitions and replication for fault tolerance in distributed mode.

Use commands to verify topic details and cluster setup.

Practice

1. What is the main purpose of running Kafka Connect in standalone mode?

easy

A. To run multiple workers for high scalability

B. To run Kafka Connect in a single process for simple or testing use

C. To enable fault tolerance across multiple machines

D. To connect Kafka with distributed databases automatically

Solution

1. Step 1: Understand standalone mode context

   Standalone mode runs Kafka Connect in one process, suitable for simple or testing scenarios.

2. Step 2: Compare with distributed mode

   Distributed mode runs multiple workers for scalability and fault tolerance, unlike standalone mode.

3. Final Answer:

   To run Kafka Connect in a single process for simple or testing use -> Option B

4. Quick Check:

   Standalone mode = single process for testing [OK]

Hint: Standalone = single process, simple use [OK]

Common Mistakes:

• Confusing standalone with distributed mode
• Thinking standalone supports fault tolerance
• Assuming standalone is for large scale pipelines

2. Which of the following is the correct command to start Kafka Connect in distributed mode?

easy

A. kafka-connect.sh start-standalone

B. connect-standalone.sh config/connect-standalone.properties

C. kafka-connect.sh start-distributed

D. connect-distributed.sh config/connect-distributed.properties

Solution

1. Step 1: Identify the script for distributed mode

   The script connect-distributed.sh is used to start Kafka Connect in distributed mode.

2. Step 2: Confirm the configuration file

   The configuration file config/connect-distributed.properties is the standard config for distributed mode.

3. Final Answer:

   connect-distributed.sh config/connect-distributed.properties -> Option D

4. Quick Check:

   Distributed mode uses connect-distributed.sh [OK]

Hint: Distributed mode uses connect-distributed.sh script [OK]

Common Mistakes:

• Using connect-standalone.sh for distributed mode
• Mixing config files between modes
• Assuming kafka-connect.sh is a valid command

3. Given the following scenario: You start Kafka Connect with connect-standalone.sh and add two connectors. What will happen if the process crashes?

medium

A. Both connectors will automatically restart on another worker

B. Connectors will continue running without interruption

C. The connectors will stop and require manual restart

D. Kafka Connect will switch to distributed mode automatically

Solution

1. Step 1: Understand standalone mode fault tolerance

   Standalone mode runs in a single process without fault tolerance or automatic recovery.

2. Step 2: Consequence of process crash

   If the process crashes, connectors stop and need manual restart because no other worker is available.

3. Final Answer:

   The connectors will stop and require manual restart -> Option C

4. Quick Check:

   Standalone mode lacks automatic recovery [OK]

Hint: Standalone mode needs manual restart after crash [OK]

Common Mistakes:

• Assuming automatic failover in standalone mode
• Thinking distributed mode starts automatically
• Believing connectors run independently of the process

4. You configured Kafka Connect in distributed mode but notice that only one worker is running and no failover happens. What is the likely cause?

medium

A. You started Kafka Connect with connect-standalone.sh instead of connect-distributed.sh

B. You did not configure the Kafka cluster correctly

C. You forgot to add connectors to the worker

D. Distributed mode does not support multiple workers

Solution

1. Step 1: Check the startup script used

   If connect-standalone.sh is used, only one worker runs without failover, even if config is for distributed mode.

2. Step 2: Understand distributed mode requirements

   Distributed mode requires starting with connect-distributed.sh to run multiple workers and enable failover.

3. Final Answer:

   You started Kafka Connect with connect-standalone.sh instead of connect-distributed.sh -> Option A

4. Quick Check:

   Wrong startup script causes single worker [OK]

Hint: Use connect-distributed.sh to run multiple workers [OK]

Common Mistakes:

• Assuming config alone enables distributed mode
• Believing distributed mode runs with standalone script
• Ignoring startup script importance

5. You have a data pipeline that must handle high volume and remain available even if one worker fails. Which Kafka Connect mode should you choose and why?

hard

A. Distributed mode, because it supports multiple workers and fault tolerance

B. Standalone mode, because it is simpler to configure

C. Standalone mode, because it automatically restarts connectors on failure

D. Distributed mode, because it runs connectors in a single process

Solution

1. Step 1: Analyze pipeline requirements

   High volume and availability require scalability and fault tolerance.

2. Step 2: Match requirements to Kafka Connect modes

   Distributed mode runs multiple workers and supports fault tolerance, meeting these needs.

3. Final Answer:

   Distributed mode, because it supports multiple workers and fault tolerance -> Option A

4. Quick Check:

   High volume + fault tolerance = distributed mode [OK]

Hint: Choose distributed mode for scalability and fault tolerance [OK]

Common Mistakes:

• Choosing standalone for high availability
• Thinking standalone auto-restarts connectors
• Misunderstanding distributed mode runs single process