Replication factor in Kafka - Mini Project: Build & Apply

Kafka Topic Replication Factor Setup

📖 Scenario: You are setting up a Kafka topic for a small business application. To ensure data safety, you want to configure the replication factor properly.

🎯 Goal: Create a Kafka topic named orders with 3 partitions and a replication factor of 2. Then verify the topic configuration.

📋 What You'll Learn

Create a Kafka topic named orders with 3 partitions and replication factor 2

Store the topic name in a variable called TOPIC_NAME

Use the kafka-topics.sh command to create the topic with the specified replication factor

Use the kafka-topics.sh command to describe the topic and verify the replication factor

Print the output of the topic description

💡 Why This Matters

🌍 Real World

Kafka topics with replication factor ensure data is copied across multiple servers. This protects against data loss if one server fails.

💼 Career

Kafka administrators and DevOps engineers often configure replication factors to balance data safety and resource use in production systems.

1

Set the Kafka topic name

Create a variable called TOPIC_NAME and set it to the string orders.

Kafka

# Create a variable called TOPIC_NAME and set it to 'orders'
# Your code here

Hint

Use the syntax VAR_NAME="value" to create a variable in bash.

2

Set the number of partitions and replication factor

Create two variables: PARTITIONS set to 3 and REPLICATION_FACTOR set to 2.

Kafka

TOPIC_NAME="orders"
# Create PARTITIONS=3 and REPLICATION_FACTOR=2
# Your code here

Hint

Use the syntax VAR_NAME=value without spaces for integer variables.

3

Create the Kafka topic with replication factor

Use the kafka-topics.sh command with --create, --topic $TOPIC_NAME, --partitions $PARTITIONS, and --replication-factor $REPLICATION_FACTOR to create the topic.

Kafka

TOPIC_NAME="orders"
PARTITIONS=3
REPLICATION_FACTOR=2
# Create the Kafka topic using kafka-topics.sh
# Your code here

Hint

Use --bootstrap-server localhost:9092 to connect to the Kafka server.

4

Describe the topic and print the replication factor

Use the kafka-topics.sh --describe --topic $TOPIC_NAME --bootstrap-server localhost:9092 command and print the output to verify the replication factor.

Kafka

TOPIC_NAME="orders"
PARTITIONS=3
REPLICATION_FACTOR=2
kafka-topics.sh --create --topic "$TOPIC_NAME" --partitions "$PARTITIONS" --replication-factor "$REPLICATION_FACTOR" --bootstrap-server localhost:9092
# Describe the topic and print the output
# Your code here

Hint

The --describe command shows topic details including replication factor.

Practice

1.

What does the replication factor in Kafka control?

easy

A. The size limit of each partition

B. The number of copies of each partition stored across brokers

C. The number of partitions in a topic

D. The number of consumers in a consumer group

Solution

1. Step 1: Understand replication factor meaning

   The replication factor defines how many copies of each partition Kafka keeps on different brokers.

2. Step 2: Differentiate from other Kafka settings

   Partitions count and consumer group size are unrelated to replication factor.

3. Final Answer:

   The number of copies of each partition stored across brokers -> Option B

4. Quick Check:

   Replication factor = number of copies [OK]

Hint: Replication factor = copies of data stored [OK]

Common Mistakes:

• Confusing replication factor with partition count
• Thinking replication factor controls consumer groups
• Assuming replication factor limits partition size

2.

Which of the following is the correct way to set the replication factor when creating a Kafka topic using the command line?

kafka-topics.sh --create --topic my-topic --partitions 3 --replication-factor ? --bootstrap-server localhost:9092

easy

A. --replication-factor 3

B. --replication-factor=3

C. --replication_factor 3

D. --replicationFactor 3

Solution

1. Step 1: Check Kafka CLI syntax for replication factor

   The correct flag is --replication-factor with a space before the value.

2. Step 2: Identify incorrect options

   Options with underscores or camelCase are invalid flags; equals sign is not used here.

3. Final Answer:

   --replication-factor 3 -> Option A

4. Quick Check:

   Use double dash and hyphen for replication factor [OK]

Hint: Use double dash and hyphen for flags in Kafka CLI [OK]

Common Mistakes:

• Using underscore instead of hyphen
• Using equals sign instead of space
• Using camelCase flag names

3.

Given a Kafka topic created with --partitions 2 and --replication-factor 3, how many total replicas will Kafka store across the cluster?

medium

A. 6

B. 3

C. 5

D. 2

Solution

1. Step 1: Calculate replicas per partition

   Each partition has 3 replicas because replication factor is 3.

2. Step 2: Multiply by number of partitions

   2 partitions x 3 replicas each = 6 total replicas.

3. Final Answer:

   6 -> Option A

4. Quick Check:

   Partitions x Replication factor = Total replicas [OK]

Hint: Multiply partitions by replication factor [OK]

Common Mistakes:

• Confusing replication factor with total partitions
• Adding instead of multiplying partitions and replicas
• Ignoring replication factor in calculation

4.

What is wrong with this Kafka topic creation command?

kafka-topics.sh --create --topic logs --partitions 3 --replication-factor 5 --bootstrap-server localhost:9092

Assuming the Kafka cluster has only 3 brokers.

medium

A. Partitions must be equal to replication factor

B. Topic name cannot be 'logs'

C. Bootstrap server address is incorrect

D. Replication factor cannot be greater than the number of brokers

Solution

1. Step 1: Understand replication factor limits

   Replication factor must be less than or equal to the number of brokers to ensure each replica is on a different broker.

2. Step 2: Check cluster size vs replication factor

   Replication factor 5 is greater than 3 brokers, which is invalid.

3. Final Answer:

   Replication factor cannot be greater than the number of brokers -> Option D

4. Quick Check:

   Replication factor ≤ brokers [OK]

Hint: Replication factor must not exceed broker count [OK]

Common Mistakes:

• Thinking partitions must equal replication factor
• Ignoring cluster size limits
• Assuming topic names have restrictions here

5.

You want to create a Kafka topic with 4 partitions and a replication factor of 3 for high availability. However, your cluster has only 2 brokers. What is the best solution?

hard

A. Create the topic with 1 partition and replication factor 3

B. Set replication factor to 4 to match partitions

C. Increase the number of brokers to at least 3

D. Create the topic with replication factor 2 instead

Solution

1. Step 1: Check replication factor vs brokers

   Replication factor 3 requires at least 3 brokers to store replicas on different machines.

2. Step 2: Evaluate options

   Reducing replication factor lowers availability; increasing brokers meets requirement.

3. Final Answer:

   Increase the number of brokers to at least 3 -> Option C

4. Quick Check:

   Replication factor ≤ brokers for high availability [OK]

Hint: Match replication factor to broker count for safety [OK]

Common Mistakes:

• Setting replication factor higher than brokers
• Reducing partitions instead of brokers
• Ignoring availability requirements