Kafka installation and setup - Mini Project: Build & Apply

Kafka Installation and Setup

📖 Scenario: You are setting up Apache Kafka on your local machine to start learning how to use it for messaging between applications.This project guides you through the basic installation and setup steps to get Kafka running.

🎯 Goal: Install Kafka, configure basic settings, start the Kafka server, and verify it is running correctly.

📋 What You'll Learn

Download and extract Kafka binaries

Set up environment variables for Kafka

Configure Kafka server properties

Start Zookeeper and Kafka server

Verify Kafka server is running

💡 Why This Matters

🌍 Real World

Kafka is widely used in real-world applications to handle real-time data streams and messaging between services.

💼 Career

Knowing how to install and configure Kafka is essential for roles in DevOps, data engineering, and backend development.

1

Download and Extract Kafka

Download the Kafka binary archive from the official Apache Kafka website and extract it to a folder called kafka_2.13-3.5.1. Then create a variable called kafka_dir and set it to the path "./kafka_2.13-3.5.1".

Kafka

# Create a variable called kafka_dir and set it to "./kafka_2.13-3.5.1"
# Your code here

Need a hint?

Use a string variable named kafka_dir to store the Kafka folder path.

2

Set Environment Variables

Create a variable called zookeeper_config and set it to the path "{kafka_dir}/config/zookeeper.properties". Also create a variable called server_config and set it to the path "{kafka_dir}/config/server.properties". Use f-strings to include kafka_dir in the paths.

Kafka

kafka_dir = "./kafka_2.13-3.5.1"
# Create zookeeper_config and server_config variables with correct paths
# Your code here

Need a hint?

Use f-strings to build the full paths for Zookeeper and Kafka server config files.

3

Start Zookeeper and Kafka Server

Use the subprocess module to run the commands to start Zookeeper and Kafka server. Create a variable called start_zookeeper_cmd with the command list ["bin/zookeeper-server-start.sh", zookeeper_config]. Create a variable called start_kafka_cmd with the command list ["bin/kafka-server-start.sh", server_config]. Then use subprocess.Popen to start Zookeeper and Kafka server using these commands with the current working directory set to kafka_dir. Assign the process objects to variables zookeeper_process and kafka_process respectively.

Kafka

import subprocess

kafka_dir = "./kafka_2.13-3.5.1"
zookeeper_config = f"{kafka_dir}/config/zookeeper.properties"
server_config = f"{kafka_dir}/config/server.properties"

# Create start_zookeeper_cmd and start_kafka_cmd lists
# Use subprocess.Popen to start both processes
# Your code here

Need a hint?

Use subprocess.Popen with cwd=kafka_dir to start the servers.

4

Verify Kafka Server is Running

Print the message "Kafka server started successfully" to confirm the setup.

Kafka

import subprocess

kafka_dir = "./kafka_2.13-3.5.1"
zookeeper_config = f"{kafka_dir}/config/zookeeper.properties"
server_config = f"{kafka_dir}/config/server.properties"

start_zookeeper_cmd = ["bin/zookeeper-server-start.sh", zookeeper_config]
start_kafka_cmd = ["bin/kafka-server-start.sh", server_config]

zookeeper_process = subprocess.Popen(start_zookeeper_cmd, cwd=kafka_dir)
kafka_process = subprocess.Popen(start_kafka_cmd, cwd=kafka_dir)

# Print confirmation message
# Your code here

Need a hint?

Use print("Kafka server started successfully") to show the confirmation.