Kafka installation and setup - Step-by-Step Execution

Process Flow - Kafka installation and setup

Download Kafka

↓

Extract Files

↓

Start Zookeeper

↓

Start Kafka Server

↓

Create Topics

↓

Produce and Consume Messages

↓

Shutdown Kafka and Zookeeper

This flow shows the main steps to install and set up Kafka: download, extract, start Zookeeper, start Kafka, create topics, use Kafka, then shutdown.

Execution Sample

Kafka

wget https://downloads.apache.org/kafka/3.5.1/kafka_2.13-3.5.1.tgz

tar -xzf kafka_2.13-3.5.1.tgz

cd kafka_2.13-3.5.1

bin/zookeeper-server-start.sh config/zookeeper.properties

bin/kafka-server-start.sh config/server.properties

This code downloads Kafka, extracts it, starts Zookeeper, then starts the Kafka server.

Process Table

Step	Action	Command Run	Result	Next Step
1	Download Kafka archive	wget https://downloads.apache.org/kafka/3.5.1/kafka_2.13-3.5.1.tgz	Kafka archive downloaded	Extract Files
2	Extract Kafka files	tar -xzf kafka_2.13-3.5.1.tgz	Kafka files extracted to folder kafka_2.13-3.5.1	Change Directory
3	Change directory to Kafka folder	cd kafka_2.13-3.5.1	Current directory is kafka_2.13-3.5.1	Start Zookeeper
4	Start Zookeeper service	bin/zookeeper-server-start.sh config/zookeeper.properties	Zookeeper started and running	Start Kafka Server
5	Start Kafka server	bin/kafka-server-start.sh config/server.properties	Kafka server started and running	Create Topics
6	Create a Kafka topic	bin/kafka-topics.sh --create --topic test --bootstrap-server localhost:9092	Topic 'test' created	Produce Messages
7	Produce messages to topic	bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9092	Ready to accept messages	Consume Messages
8	Consume messages from topic	bin/kafka-console-consumer.sh --topic test --from-beginning --bootstrap-server localhost:9092	Messages received and displayed	Shutdown
9	Shutdown Kafka server	Ctrl+C or kill process	Kafka server stopped	Shutdown Zookeeper
10	Shutdown Zookeeper	Ctrl+C or kill process	Zookeeper stopped	End

💡 All services stopped, Kafka installation and setup complete

Status Tracker

Variable/Service	Start	After Step 4	After Step 5	After Step 9	After Step 10
Zookeeper	Not running	Running	Running	Running	Stopped
Kafka Server	Not running	Not running	Running	Stopped	Stopped
Kafka Topic 'test'	Not created	Not created	Created	Created	Created

Key Moments - 3 Insights

Why do we need to start Zookeeper before Kafka server?

Kafka depends on Zookeeper to manage cluster metadata. As shown in execution_table step 4, Zookeeper must be running before starting Kafka server in step 5.

What happens if we try to create a topic before starting Kafka server?

The topic creation will fail because Kafka server is not running to accept commands. Execution_table shows topic creation only after Kafka server starts at step 6.

How do we stop Kafka and Zookeeper services safely?

Use Ctrl+C or kill the processes as in steps 9 and 10. This cleanly stops the services to avoid data loss.

Visual Quiz - 3 Questions

Look at the execution_table, at which step does Kafka server start running?

A Step 5

B Step 4

C Step 6

D Step 7

💡 Hint

Check the 'Result' column for when Kafka server status changes to running.

According to variable_tracker, what is the status of Zookeeper after step 9?

A Stopped

B Not running

C Running

D Starting

💡 Hint

Look at the Zookeeper row and the column 'After Step 9' in variable_tracker.

If you skip step 4 and start Kafka server directly, what will likely happen?

A Kafka server starts normally

B Kafka server fails to start

C Topic creation succeeds

D Zookeeper starts automatically

💡 Hint

Refer to key_moments about the dependency of Kafka on Zookeeper before starting.

Concept Snapshot

Kafka Installation and Setup Quick Steps:
1. Download and extract Kafka archive.
2. Start Zookeeper service first.
3. Start Kafka server next.
4. Create topics and produce/consume messages.
5. Shutdown Kafka and Zookeeper cleanly.
Remember: Zookeeper must run before Kafka server.

Full Transcript

This visual execution shows the step-by-step process to install and set up Kafka. First, you download the Kafka archive and extract it. Then you start the Zookeeper service, which Kafka needs to manage its cluster. After Zookeeper is running, you start the Kafka server. Next, you create topics to organize messages. You can then produce messages to these topics and consume them to verify Kafka is working. Finally, you stop the Kafka server and then Zookeeper to finish the setup safely. The tables track each step's commands, results, and the state of services. Key moments clarify why Zookeeper must start first and how to stop services properly. The quiz tests understanding of these steps and dependencies.

Practice

1. What is the first step to start Kafka on your local machine?

easy

A. Start Zookeeper service

B. Create Kafka topics

C. Run Kafka producer

D. Install Java

Solution

1. Step 1: Understand Kafka dependencies

   Kafka requires Zookeeper to coordinate its services.

2. Step 2: Start Zookeeper before Kafka

   You must start Zookeeper first to allow Kafka to connect and run properly.

3. Final Answer:

   Start Zookeeper service -> Option A

4. Quick Check:

   Kafka needs Zookeeper first = Start Zookeeper service [OK]

Hint: Always start Zookeeper before Kafka server [OK]

Common Mistakes:

• Starting Kafka before Zookeeper
• Skipping Zookeeper start
• Trying to create topics before server runs

2. Which command correctly starts the Kafka server after Zookeeper is running?

easy

A. bin/kafka-topics.sh --create --topic test

B. bin/zookeeper-server-start.sh config/zookeeper.properties

C. bin/kafka-server-start.sh config/server.properties

D. bin/kafka-console-producer.sh --topic test

Solution

1. Step 1: Identify Kafka server start command

   The Kafka server is started using kafka-server-start.sh with server properties.

2. Step 2: Differentiate from other commands

   Zookeeper start uses zookeeper-server-start.sh; topic creation and producer commands are different tools.

3. Final Answer:

   bin/kafka-server-start.sh config/server.properties -> Option C

4. Quick Check:

   Kafka server start = bin/kafka-server-start.sh config/server.properties [OK]

Hint: Kafka server uses kafka-server-start.sh script [OK]

Common Mistakes:

• Using zookeeper start command for Kafka
• Confusing topic creation with server start
• Running producer before server

3. What will be the output when you run this command after starting Kafka server?
bin/kafka-topics.sh --list --bootstrap-server localhost:9092

medium

A. Lists all existing Kafka topics

B. Starts the Kafka server

C. Creates a new topic named 'localhost'

D. Shows Zookeeper status

Solution

1. Step 1: Understand the command purpose

   The command lists topics using the Kafka bootstrap server address.

2. Step 2: Identify output type

   It outputs all existing topics in Kafka, not starting server or creating topics.

3. Final Answer:

   Lists all existing Kafka topics -> Option A

4. Quick Check:

   --list shows topics = Lists all existing Kafka topics [OK]

Hint: Use --list with --bootstrap-server to see topics [OK]

Common Mistakes:

• Thinking it starts Kafka server
• Assuming it creates topics
• Confusing with Zookeeper commands

4. You tried to start Kafka server but got an error saying 'Could not connect to Zookeeper'. What is the likely cause?

medium

A. Kafka server is already running

B. Zookeeper service is not running

C. Topic name is missing

D. Java is not installed

Solution

1. Step 1: Analyze error message

   'Could not connect to Zookeeper' means Kafka cannot reach Zookeeper service.

2. Step 2: Identify cause

   This usually happens if Zookeeper is not started or crashed.

3. Final Answer:

   Zookeeper service is not running -> Option B

4. Quick Check:

   Connection error = Zookeeper not running [OK]

Hint: Check Zookeeper is running before Kafka start [OK]

Common Mistakes:

• Ignoring Zookeeper status
• Assuming Kafka server conflict
• Blaming missing topic name

5. You want to test Kafka by sending messages to a topic named 'test'. Which sequence of commands is correct after starting Zookeeper and Kafka server?

hard

A. Start kafka-console-consumer.sh, then create topic with kafka-topics.sh

B. Only start Zookeeper and produce messages directly

C. Run kafka-server-start.sh twice, then produce messages

D. Create topic with kafka-topics.sh, then produce messages with kafka-console-producer.sh

Solution

1. Step 1: Create the topic first

   You must create the topic 'test' before sending messages to it.

2. Step 2: Use producer to send messages

   After topic creation, use kafka-console-producer.sh to send messages to 'test'.

3. Final Answer:

   Create topic with kafka-topics.sh, then produce messages with kafka-console-producer.sh -> Option D

4. Quick Check:

   Create topic then produce = Create topic with kafka-topics.sh, then produce messages with kafka-console-producer.sh [OK]

Hint: Create topic before producing messages [OK]

Common Mistakes:

• Producing messages before topic creation
• Starting Kafka server twice unnecessarily
• Skipping Kafka server start