Kafka installation and setup - Commands & Configuration

Introduction

Kafka is a tool that helps you send and receive messages between different parts of your system quickly and reliably. Installing and setting it up correctly lets you start using it to handle data streams in real time.

When you want to collect logs from many servers and process them centrally.

When you need to build a system that reacts instantly to new data, like notifications or alerts.

When you want to connect different applications so they can share information without being directly linked.

When you want to store data streams for later analysis or replay.

When you need a reliable way to move large amounts of data between systems.

Config File - server.properties

server.properties

broker.id=1
listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000

This file configures the Kafka broker settings:

• broker.id: Unique ID for this Kafka server.
• listeners: Network address and port Kafka listens on.
• log.dirs: Folder where Kafka stores messages.
• num.network.threads and num.io.threads: Threads for handling network and disk I/O.
• socket.*: Network buffer sizes.
• log.retention.hours: How long messages are kept.
• zookeeper.connect: Address of Zookeeper managing Kafka cluster.

Commands

Updates the list of available packages and their versions on your system.

Terminal

sudo apt-get update

Expected Output

Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease

Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]

Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB]

Fetched 215 kB in 1s (234 kB/s)

Reading package lists... Done

Installs Java runtime needed to run Kafka.

Terminal

sudo apt-get install -y openjdk-11-jre-headless

Expected Output

Reading package lists... Done

Building dependency tree       

Reading state information... Done

The following NEW packages will be installed:

  openjdk-11-jre-headless

0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.

Need to get 36.3 MB of archives.

After this operation, 114 MB of additional disk space will be used.

Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 openjdk-11-jre-headless amd64 11.0.11+9-0ubuntu2~20.04 [36.3 MB]

Fetched 36.3 MB in 3s (11.9 MB/s)

Selecting previously unselected package openjdk-11-jre-headless:amd64.

(Reading database ... 123456 files and directories currently installed.)

Preparing to unpack .../openjdk-11-jre-headless_11.0.11+9-0ubuntu2~20.04_amd64.deb ...

Unpacking openjdk-11-jre-headless:amd64 (11.0.11+9-0ubuntu2~20.04) ...

Setting up openjdk-11-jre-headless:amd64 (11.0.11+9-0ubuntu2~20.04) ...

→

-y - Automatically answer yes to prompts

Downloads the Kafka software package version 3.4.0.

Terminal

wget https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz

Expected Output

--2024-06-01 12:00:00--  https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz

Resolving downloads.apache.org (downloads.apache.org)... 104.16.0.30, 104.16.1.30

Connecting to downloads.apache.org (downloads.apache.org)|104.16.0.30|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 123456789 (118M) [application/x-gzip]

Saving to: 'kafka_2.13-3.4.0.tgz'



kafka_2.13-3.4.0.tgz 100%[===================>] 117.81M  5.12MB/s    in 24s     



2024-06-01 12:00:24 (4.91 MB/s) - 'kafka_2.13-3.4.0.tgz' saved [123456789/123456789]

Extracts the Kafka package files from the downloaded archive.

Terminal

tar -xzf kafka_2.13-3.4.0.tgz

Expected Output

No output (command runs silently)

→

-xzf - Extract gzip compressed tar archive

Starts the Zookeeper service in the background, which Kafka needs to manage its cluster.

Terminal

cd kafka_2.13-3.4.0 && bin/zookeeper-server-start.sh config/zookeeper.properties &

Expected Output

[2024-06-01 12:01:00,000] INFO Starting zookeeper server ...

[2024-06-01 12:01:01,000] INFO binding to port 0.0.0.0/0.0.0.0:2181

[2024-06-01 12:01:01,500] INFO zookeeper state change: LATENT -> CONNECTING

[2024-06-01 12:01:02,000] INFO zookeeper state change: CONNECTING -> CONNECTED

Starts the Kafka broker service in the background using the configuration file.

Terminal

bin/kafka-server-start.sh config/server.properties &

Expected Output

[2024-06-01 12:01:10,000] INFO Kafka version : 3.4.0

[2024-06-01 12:01:10,100] INFO Kafka started (kafka.server.KafkaServer)

Creates a new topic named 'test-topic' on the Kafka server to send and receive messages.

Terminal

bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Expected Output

Created topic test-topic.

→

--create - Create a new topic

→

--bootstrap-server - Kafka server address

Lists all topics currently available on the Kafka server to verify the new topic was created.

Terminal

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Expected Output

test-topic

→

--list - List all topics

Key Concept

If you remember nothing else from this pattern, remember: Kafka needs Java and Zookeeper running before starting the Kafka broker.

Common Mistakes

Trying to start Kafka broker before starting Zookeeper

Kafka depends on Zookeeper to manage cluster metadata, so it will fail to start without it.

Always start Zookeeper first, then start the Kafka broker.

Not setting the correct broker.id or port in server.properties

Kafka broker needs a unique ID and correct network settings to work properly and avoid conflicts.

Set broker.id to a unique number and listeners to the correct port before starting Kafka.

Not verifying topic creation after running the create command

You might think the topic was created but it could have failed silently or due to misconfiguration.

Always run the list topics command to confirm your topic exists.

Summary

Update your system and install Java to prepare for Kafka installation.

Download and extract Kafka, then start Zookeeper and Kafka broker services.

Create and verify Kafka topics to start sending and receiving messages.

Practice

1. What is the first step to start Kafka on your local machine?

easy

A. Start Zookeeper service

B. Create Kafka topics

C. Run Kafka producer

D. Install Java

Solution

1. Step 1: Understand Kafka dependencies

   Kafka requires Zookeeper to coordinate its services.

2. Step 2: Start Zookeeper before Kafka

   You must start Zookeeper first to allow Kafka to connect and run properly.

3. Final Answer:

   Start Zookeeper service -> Option A

4. Quick Check:

   Kafka needs Zookeeper first = Start Zookeeper service [OK]

Hint: Always start Zookeeper before Kafka server [OK]

Common Mistakes:

• Starting Kafka before Zookeeper
• Skipping Zookeeper start
• Trying to create topics before server runs

2. Which command correctly starts the Kafka server after Zookeeper is running?

easy

A. bin/kafka-topics.sh --create --topic test

B. bin/zookeeper-server-start.sh config/zookeeper.properties

C. bin/kafka-server-start.sh config/server.properties

D. bin/kafka-console-producer.sh --topic test

Solution

1. Step 1: Identify Kafka server start command

   The Kafka server is started using kafka-server-start.sh with server properties.

2. Step 2: Differentiate from other commands

   Zookeeper start uses zookeeper-server-start.sh; topic creation and producer commands are different tools.

3. Final Answer:

   bin/kafka-server-start.sh config/server.properties -> Option C

4. Quick Check:

   Kafka server start = bin/kafka-server-start.sh config/server.properties [OK]

Hint: Kafka server uses kafka-server-start.sh script [OK]

Common Mistakes:

• Using zookeeper start command for Kafka
• Confusing topic creation with server start
• Running producer before server

3. What will be the output when you run this command after starting Kafka server?
bin/kafka-topics.sh --list --bootstrap-server localhost:9092

medium

A. Lists all existing Kafka topics

B. Starts the Kafka server

C. Creates a new topic named 'localhost'

D. Shows Zookeeper status

Solution

1. Step 1: Understand the command purpose

   The command lists topics using the Kafka bootstrap server address.

2. Step 2: Identify output type

   It outputs all existing topics in Kafka, not starting server or creating topics.

3. Final Answer:

   Lists all existing Kafka topics -> Option A

4. Quick Check:

   --list shows topics = Lists all existing Kafka topics [OK]

Hint: Use --list with --bootstrap-server to see topics [OK]

Common Mistakes:

• Thinking it starts Kafka server
• Assuming it creates topics
• Confusing with Zookeeper commands

4. You tried to start Kafka server but got an error saying 'Could not connect to Zookeeper'. What is the likely cause?

medium

A. Kafka server is already running

B. Zookeeper service is not running

C. Topic name is missing

D. Java is not installed

Solution

1. Step 1: Analyze error message

   'Could not connect to Zookeeper' means Kafka cannot reach Zookeeper service.

2. Step 2: Identify cause

   This usually happens if Zookeeper is not started or crashed.

3. Final Answer:

   Zookeeper service is not running -> Option B

4. Quick Check:

   Connection error = Zookeeper not running [OK]

Hint: Check Zookeeper is running before Kafka start [OK]

Common Mistakes:

• Ignoring Zookeeper status
• Assuming Kafka server conflict
• Blaming missing topic name

5. You want to test Kafka by sending messages to a topic named 'test'. Which sequence of commands is correct after starting Zookeeper and Kafka server?

hard

A. Start kafka-console-consumer.sh, then create topic with kafka-topics.sh

B. Only start Zookeeper and produce messages directly

C. Run kafka-server-start.sh twice, then produce messages

D. Create topic with kafka-topics.sh, then produce messages with kafka-console-producer.sh

Solution

1. Step 1: Create the topic first

   You must create the topic 'test' before sending messages to it.

2. Step 2: Use producer to send messages

   After topic creation, use kafka-console-producer.sh to send messages to 'test'.

3. Final Answer:

   Create topic with kafka-topics.sh, then produce messages with kafka-console-producer.sh -> Option D

4. Quick Check:

   Create topic then produce = Create topic with kafka-topics.sh, then produce messages with kafka-console-producer.sh [OK]

Hint: Create topic before producing messages [OK]

Common Mistakes:

• Producing messages before topic creation
• Starting Kafka server twice unnecessarily
• Skipping Kafka server start