Introduction

Kafka is a tool that helps you send and receive messages between different parts of your system quickly and reliably. Installing and setting it up correctly lets you start using it to handle data streams in real time.

When you want to collect logs from many servers and process them centrally.

When you need to build a system that reacts instantly to new data, like notifications or alerts.

When you want to connect different applications so they can share information without being directly linked.

When you want to store data streams for later analysis or replay.

When you need a reliable way to move large amounts of data between systems.

Config File - server.properties

server.properties

broker.id=1
listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000

This file configures the Kafka broker settings:

broker.id: Unique ID for this Kafka server.
listeners: Network address and port Kafka listens on.
log.dirs: Folder where Kafka stores messages.
num.network.threads and num.io.threads: Threads for handling network and disk I/O.
socket.*: Network buffer sizes.
log.retention.hours: How long messages are kept.
zookeeper.connect: Address of Zookeeper managing Kafka cluster.

Commands

Updates the list of available packages and their versions on your system.

Terminal

sudo apt-get update

Expected OutputExpected

Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [101 kB] Fetched 215 kB in 1s (234 kB/s) Reading package lists... Done

Installs Java runtime needed to run Kafka.

Terminal

sudo apt-get install -y openjdk-11-jre-headless

Expected OutputExpected

Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: openjdk-11-jre-headless 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 36.3 MB of archives. After this operation, 114 MB of additional disk space will be used. Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 openjdk-11-jre-headless amd64 11.0.11+9-0ubuntu2~20.04 [36.3 MB] Fetched 36.3 MB in 3s (11.9 MB/s) Selecting previously unselected package openjdk-11-jre-headless:amd64. (Reading database ... 123456 files and directories currently installed.) Preparing to unpack .../openjdk-11-jre-headless_11.0.11+9-0ubuntu2~20.04_amd64.deb ... Unpacking openjdk-11-jre-headless:amd64 (11.0.11+9-0ubuntu2~20.04) ... Setting up openjdk-11-jre-headless:amd64 (11.0.11+9-0ubuntu2~20.04) ...

→

-y - Automatically answer yes to prompts

Downloads the Kafka software package version 3.4.0.

Terminal

wget https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz

Expected OutputExpected

--2024-06-01 12:00:00-- https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz Resolving downloads.apache.org (downloads.apache.org)... 104.16.0.30, 104.16.1.30 Connecting to downloads.apache.org (downloads.apache.org)|104.16.0.30|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 123456789 (118M) [application/x-gzip] Saving to: 'kafka_2.13-3.4.0.tgz' kafka_2.13-3.4.0.tgz 100%[===================>] 117.81M 5.12MB/s in 24s 2024-06-01 12:00:24 (4.91 MB/s) - 'kafka_2.13-3.4.0.tgz' saved [123456789/123456789]

Extracts the Kafka package files from the downloaded archive.

Terminal

tar -xzf kafka_2.13-3.4.0.tgz

Expected OutputExpected

No output (command runs silently)

→

-xzf - Extract gzip compressed tar archive

Starts the Zookeeper service in the background, which Kafka needs to manage its cluster.

Terminal

cd kafka_2.13-3.4.0 && bin/zookeeper-server-start.sh config/zookeeper.properties &

Expected OutputExpected

[2024-06-01 12:01:00,000] INFO Starting zookeeper server ... [2024-06-01 12:01:01,000] INFO binding to port 0.0.0.0/0.0.0.0:2181 [2024-06-01 12:01:01,500] INFO zookeeper state change: LATENT -> CONNECTING [2024-06-01 12:01:02,000] INFO zookeeper state change: CONNECTING -> CONNECTED

Starts the Kafka broker service in the background using the configuration file.

Terminal

bin/kafka-server-start.sh config/server.properties &

Expected OutputExpected

[2024-06-01 12:01:10,000] INFO Kafka version : 3.4.0 [2024-06-01 12:01:10,100] INFO Kafka started (kafka.server.KafkaServer)

Creates a new topic named 'test-topic' on the Kafka server to send and receive messages.

Terminal

bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

Expected OutputExpected

Created topic test-topic.

→

--create - Create a new topic

→

--bootstrap-server - Kafka server address

Lists all topics currently available on the Kafka server to verify the new topic was created.

Terminal

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Expected OutputExpected

test-topic

→

--list - List all topics

Key Concept

If you remember nothing else from this pattern, remember: Kafka needs Java and Zookeeper running before starting the Kafka broker.

Common Mistakes

Trying to start Kafka broker before starting Zookeeper

Kafka depends on Zookeeper to manage cluster metadata, so it will fail to start without it.

Always start Zookeeper first, then start the Kafka broker.

Not setting the correct broker.id or port in server.properties

Kafka broker needs a unique ID and correct network settings to work properly and avoid conflicts.

Set broker.id to a unique number and listeners to the correct port before starting Kafka.

Not verifying topic creation after running the create command

You might think the topic was created but it could have failed silently or due to misconfiguration.

Always run the list topics command to confirm your topic exists.

Summary

Update your system and install Java to prepare for Kafka installation.

Download and extract Kafka, then start Zookeeper and Kafka broker services.

Create and verify Kafka topics to start sending and receiving messages.