Kafka installation and setup - Mini Project: Build & Apply

Kafka Installation and Setup

📖 Scenario: You are setting up Apache Kafka on your local machine to start learning how to use it for messaging between applications.This project guides you through the basic installation and setup steps to get Kafka running.

🎯 Goal: Install Kafka, configure basic settings, start the Kafka server, and verify it is running correctly.

📋 What You'll Learn

Download and extract Kafka binaries

Set up environment variables for Kafka

Configure Kafka server properties

Start Zookeeper and Kafka server

Verify Kafka server is running

💡 Why This Matters

🌍 Real World

Kafka is widely used in real-world applications to handle real-time data streams and messaging between services.

💼 Career

Knowing how to install and configure Kafka is essential for roles in DevOps, data engineering, and backend development.

1

Download and Extract Kafka

Download the Kafka binary archive from the official Apache Kafka website and extract it to a folder called kafka_2.13-3.5.1. Then create a variable called kafka_dir and set it to the path "./kafka_2.13-3.5.1".

Kafka

# Create a variable called kafka_dir and set it to "./kafka_2.13-3.5.1"
# Your code here

Hint

Use a string variable named kafka_dir to store the Kafka folder path.

2

Set Environment Variables

Create a variable called zookeeper_config and set it to the path "{kafka_dir}/config/zookeeper.properties". Also create a variable called server_config and set it to the path "{kafka_dir}/config/server.properties". Use f-strings to include kafka_dir in the paths.

Kafka

kafka_dir = "./kafka_2.13-3.5.1"
# Create zookeeper_config and server_config variables with correct paths
# Your code here

Hint

Use f-strings to build the full paths for Zookeeper and Kafka server config files.

3

Start Zookeeper and Kafka Server

Use the subprocess module to run the commands to start Zookeeper and Kafka server. Create a variable called start_zookeeper_cmd with the command list ["bin/zookeeper-server-start.sh", zookeeper_config]. Create a variable called start_kafka_cmd with the command list ["bin/kafka-server-start.sh", server_config]. Then use subprocess.Popen to start Zookeeper and Kafka server using these commands with the current working directory set to kafka_dir. Assign the process objects to variables zookeeper_process and kafka_process respectively.

Kafka

import subprocess

kafka_dir = "./kafka_2.13-3.5.1"
zookeeper_config = f"{kafka_dir}/config/zookeeper.properties"
server_config = f"{kafka_dir}/config/server.properties"

# Create start_zookeeper_cmd and start_kafka_cmd lists
# Use subprocess.Popen to start both processes
# Your code here

Hint

Use subprocess.Popen with cwd=kafka_dir to start the servers.

4

Verify Kafka Server is Running

Print the message "Kafka server started successfully" to confirm the setup.

Kafka

import subprocess

kafka_dir = "./kafka_2.13-3.5.1"
zookeeper_config = f"{kafka_dir}/config/zookeeper.properties"
server_config = f"{kafka_dir}/config/server.properties"

start_zookeeper_cmd = ["bin/zookeeper-server-start.sh", zookeeper_config]
start_kafka_cmd = ["bin/kafka-server-start.sh", server_config]

zookeeper_process = subprocess.Popen(start_zookeeper_cmd, cwd=kafka_dir)
kafka_process = subprocess.Popen(start_kafka_cmd, cwd=kafka_dir)

# Print confirmation message
# Your code here

Hint

Use print("Kafka server started successfully") to show the confirmation.

Practice

1. What is the first step to start Kafka on your local machine?

easy

A. Start Zookeeper service

B. Create Kafka topics

C. Run Kafka producer

D. Install Java

Solution

1. Step 1: Understand Kafka dependencies

   Kafka requires Zookeeper to coordinate its services.

2. Step 2: Start Zookeeper before Kafka

   You must start Zookeeper first to allow Kafka to connect and run properly.

3. Final Answer:

   Start Zookeeper service -> Option A

4. Quick Check:

   Kafka needs Zookeeper first = Start Zookeeper service [OK]

Hint: Always start Zookeeper before Kafka server [OK]

Common Mistakes:

• Starting Kafka before Zookeeper
• Skipping Zookeeper start
• Trying to create topics before server runs

2. Which command correctly starts the Kafka server after Zookeeper is running?

easy

A. bin/kafka-topics.sh --create --topic test

B. bin/zookeeper-server-start.sh config/zookeeper.properties

C. bin/kafka-server-start.sh config/server.properties

D. bin/kafka-console-producer.sh --topic test

Solution

1. Step 1: Identify Kafka server start command

   The Kafka server is started using kafka-server-start.sh with server properties.

2. Step 2: Differentiate from other commands

   Zookeeper start uses zookeeper-server-start.sh; topic creation and producer commands are different tools.

3. Final Answer:

   bin/kafka-server-start.sh config/server.properties -> Option C

4. Quick Check:

   Kafka server start = bin/kafka-server-start.sh config/server.properties [OK]

Hint: Kafka server uses kafka-server-start.sh script [OK]

Common Mistakes:

• Using zookeeper start command for Kafka
• Confusing topic creation with server start
• Running producer before server

3. What will be the output when you run this command after starting Kafka server?
bin/kafka-topics.sh --list --bootstrap-server localhost:9092

medium

A. Lists all existing Kafka topics

B. Starts the Kafka server

C. Creates a new topic named 'localhost'

D. Shows Zookeeper status

Solution

1. Step 1: Understand the command purpose

   The command lists topics using the Kafka bootstrap server address.

2. Step 2: Identify output type

   It outputs all existing topics in Kafka, not starting server or creating topics.

3. Final Answer:

   Lists all existing Kafka topics -> Option A

4. Quick Check:

   --list shows topics = Lists all existing Kafka topics [OK]

Hint: Use --list with --bootstrap-server to see topics [OK]

Common Mistakes:

• Thinking it starts Kafka server
• Assuming it creates topics
• Confusing with Zookeeper commands

4. You tried to start Kafka server but got an error saying 'Could not connect to Zookeeper'. What is the likely cause?

medium

A. Kafka server is already running

B. Zookeeper service is not running

C. Topic name is missing

D. Java is not installed

Solution

1. Step 1: Analyze error message

   'Could not connect to Zookeeper' means Kafka cannot reach Zookeeper service.

2. Step 2: Identify cause

   This usually happens if Zookeeper is not started or crashed.

3. Final Answer:

   Zookeeper service is not running -> Option B

4. Quick Check:

   Connection error = Zookeeper not running [OK]

Hint: Check Zookeeper is running before Kafka start [OK]

Common Mistakes:

• Ignoring Zookeeper status
• Assuming Kafka server conflict
• Blaming missing topic name

5. You want to test Kafka by sending messages to a topic named 'test'. Which sequence of commands is correct after starting Zookeeper and Kafka server?

hard

A. Start kafka-console-consumer.sh, then create topic with kafka-topics.sh

B. Only start Zookeeper and produce messages directly

C. Run kafka-server-start.sh twice, then produce messages

D. Create topic with kafka-topics.sh, then produce messages with kafka-console-producer.sh

Solution

1. Step 1: Create the topic first

   You must create the topic 'test' before sending messages to it.

2. Step 2: Use producer to send messages

   After topic creation, use kafka-console-producer.sh to send messages to 'test'.

3. Final Answer:

   Create topic with kafka-topics.sh, then produce messages with kafka-console-producer.sh -> Option D

4. Quick Check:

   Create topic then produce = Create topic with kafka-topics.sh, then produce messages with kafka-console-producer.sh [OK]

Hint: Create topic before producing messages [OK]

Common Mistakes:

• Producing messages before topic creation
• Starting Kafka server twice unnecessarily
• Skipping Kafka server start