Kafka integration with Hadoop - Step-by-Step Execution

Concept Flow - Kafka integration with Hadoop

Kafka Producer sends messages

↓

Kafka Broker stores messages

↓

Hadoop Kafka Consumer reads messages

↓

Data ingested into Hadoop HDFS or processed by Hadoop ecosystem

↓

Data available for batch or real-time analytics

Data flows from Kafka producers to Kafka brokers, then Hadoop consumers read and store or process data in Hadoop.

Execution Sample

Hadoop

kafka-console-producer --topic test-topic --broker-list localhost:9092
kafka-console-consumer --topic test-topic --bootstrap-server localhost:9092
hdfs dfs -ls /kafka-data

Send messages to Kafka, consume them with Hadoop, then check data stored in HDFS.

Execution Table

Step	Action	Component	Data State	Result
1	Send message 'Hello Hadoop'	Kafka Producer	Message created	Message sent to Kafka topic
2	Store message	Kafka Broker	Message stored in topic partition	Message available for consumers
3	Consume message	Hadoop Kafka Consumer	Reads message from Kafka	Message ingested into Hadoop
4	Store in HDFS	Hadoop HDFS	Data written to HDFS path	Data available for processing
5	List HDFS directory	HDFS CLI	Reads directory contents	Shows stored Kafka data files
6	Exit	Process	No more messages	Consumption stops

💡 No more messages in Kafka topic, consumer stops reading

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
Kafka Message	None	'Hello Hadoop'	Stored in Kafka	Consumed by Hadoop	Stored in HDFS	Available for analytics
HDFS Data	Empty	Empty	Empty	Empty	Data written	Data ready

Key Moments - 3 Insights

Why does the Hadoop consumer need to connect to Zookeeper?

The Hadoop Kafka consumer uses Zookeeper to discover Kafka brokers and manage offsets, as shown in execution_table step 3.

What happens if there are no new messages in Kafka?

The consumer stops reading and exits, as shown in execution_table step 6 where no more messages cause termination.

How do we verify data is stored in Hadoop after consumption?

By listing the HDFS directory with 'hdfs dfs -ls', as shown in execution_table step 5, we confirm data presence.

Visual Quiz - 3 Questions

Look at the execution table, at which step is the message stored in Kafka?

A Step 2

B Step 1

C Step 3

D Step 4

💡 Hint

Check the 'Component' and 'Result' columns in execution_table row for Step 2.

According to variable_tracker, what is the state of 'HDFS Data' after Step 4?

A Empty

B Data written

C Consumed by Hadoop

D Available for analytics

💡 Hint

Look at the 'HDFS Data' row under 'After Step 4' in variable_tracker.

If new messages keep arriving, what happens to the consumer process?

A It stops immediately

B It restarts from the beginning

C It continues consuming messages

D It deletes old messages

💡 Hint

Refer to execution_table exit_note and step 6 about consumer stopping only when no messages remain.

Concept Snapshot

Kafka integration with Hadoop:
- Kafka producers send messages to Kafka brokers.
- Hadoop consumers read messages from Kafka using Zookeeper.
- Messages are ingested into Hadoop HDFS.
- Data is then available for batch or real-time analytics.
- Use CLI commands to produce, consume, and verify data.

Full Transcript

This visual execution shows how Kafka integrates with Hadoop. First, a Kafka producer sends a message to a Kafka topic. The Kafka broker stores this message. Then, a Hadoop Kafka consumer reads the message from Kafka using Zookeeper for coordination. The consumed message is stored in Hadoop's HDFS. We verify the data storage by listing the HDFS directory. The consumer stops when no more messages are available. Variables like 'Kafka Message' and 'HDFS Data' change state step-by-step, showing the flow of data from Kafka to Hadoop storage. Key moments clarify why Zookeeper is needed, what happens when no messages remain, and how to check data in Hadoop. The quiz tests understanding of message storage, data state in HDFS, and consumer behavior with new messages.

Practice

1. What is the main purpose of integrating Kafka with Hadoop in data science?

easy

A. To move streaming data into Hadoop storage for analysis

B. To replace Hadoop's storage with Kafka's storage

C. To run machine learning models inside Kafka

D. To visualize data directly from Kafka without storage

Solution

1. Step 1: Understand Kafka and Hadoop roles

   Kafka streams data in real-time, Hadoop stores large data for analysis.

2. Step 2: Identify integration purpose

   Integration moves streaming data from Kafka into Hadoop for big data processing.

3. Final Answer:

   To move streaming data into Hadoop storage for analysis -> Option A

4. Quick Check:

   Kafka streams + Hadoop stores = Integration purpose [OK]

Hint: Kafka streams data; Hadoop stores it for analysis [OK]

Common Mistakes:

• Thinking Kafka replaces Hadoop storage
• Assuming Kafka runs ML models
• Believing visualization happens directly in Kafka

2. Which of the following is the correct command to consume messages from a Kafka topic named sensor_data?

easy

A. kafka-run-class kafka.Consumer sensor_data

B. kafka-console-producer --topic sensor_data --bootstrap-server localhost:9092

C. kafka-console-consumer --topic sensor_data --bootstrap-server localhost:9092

D. hadoop fs -cat /sensor_data

Solution

1. Step 1: Identify Kafka consumer command

   The command to read messages is kafka-console-consumer with topic and server options.

2. Step 2: Match command to options

   kafka-console-consumer --topic sensor_data --bootstrap-server localhost:9092 uses correct consumer command and parameters for topic sensor_data.

3. Final Answer:

   kafka-console-consumer --topic sensor_data --bootstrap-server localhost:9092 -> Option C

4. Quick Check:

   Consumer command with topic and server [OK]

Hint: Consumer command starts with kafka-console-consumer [OK]

Common Mistakes:

• Using producer command instead of consumer
• Confusing Hadoop fs commands with Kafka
• Using outdated or incorrect Kafka class commands

3. Given the following Flume configuration snippet to move Kafka data to HDFS:

agent.sources = kafkaSource
agent.channels = memoryChannel
agent.sinks = hdfsSink

agent.sources.kafkaSource.type = org.apache.flume.source.kafka.KafkaSource
agent.sources.kafkaSource.kafka.bootstrap.servers = localhost:9092
agent.sources.kafkaSource.kafka.topics = sensor_data

agent.channels.memoryChannel.type = memory

agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.hdfs.path = /data/sensor
agent.sinks.hdfsSink.channel = memoryChannel

What will be the HDFS directory where Kafka data is stored?

medium

A. /data/sensor

B. /kafka/sensor_data

C. /user/flume/kafka

D. /tmp/kafka_data

Solution

1. Step 1: Locate HDFS path in sink configuration

   The sink hdfsSink has hdfs.path = /data/sensor.

2. Step 2: Confirm data storage directory

   Kafka data consumed by Flume will be stored in /data/sensor on HDFS.

3. Final Answer:

   /data/sensor -> Option A

4. Quick Check:

   HDFS sink path = /data/sensor [OK]

Hint: Check hdfs.path in sink config for storage location [OK]

Common Mistakes:

• Confusing Kafka topic name with HDFS path
• Assuming default Flume user directory
• Ignoring channel configuration

4. You set up Kafka to HDFS data transfer using Flume, but no data appears in HDFS. Which of the following is the most likely error in the Flume config?

medium

A. The memory channel type is set to file

B. The HDFS path is set correctly

C. The Flume agent name is missing

D. The Kafka topic name in kafka.topics is misspelled

Solution

1. Step 1: Check common config errors

   Misspelling Kafka topic means Flume reads no data from Kafka.

2. Step 2: Evaluate options

   Correct HDFS path and channel type do not cause no data; missing agent name causes startup error, not silent failure.

3. Final Answer:

   The Kafka topic name in kafka.topics is misspelled -> Option D

4. Quick Check:

   Wrong topic name = no data read [OK]

Hint: Check Kafka topic spelling first if no data in HDFS [OK]

Common Mistakes:

• Assuming HDFS path causes no data
• Changing channel type incorrectly
• Ignoring agent name errors

5. You want to build a pipeline that streams sensor data from Kafka into Hadoop HDFS and then runs batch analysis daily. Which combination of tools and steps is best?

hard

A. Use Kafka producer to write directly to HDFS, then run Spark streaming

B. Use KafkaSource in Flume to ingest data into HDFS, then run Hadoop MapReduce daily

C. Use Hadoop fs commands to read Kafka topics, then store in local files

D. Use Kafka consumer to write data to local disk, then upload manually to HDFS

Solution

1. Step 1: Identify streaming ingestion method

   Flume with KafkaSource is designed to stream Kafka data into HDFS efficiently.

2. Step 2: Choose batch analysis tool

   Hadoop MapReduce runs batch jobs on HDFS data daily, fitting the requirement.

3. Final Answer:

   Use KafkaSource in Flume to ingest data into HDFS, then run Hadoop MapReduce daily -> Option B

4. Quick Check:

   Flume + KafkaSource + HDFS + MapReduce = correct pipeline [OK]

Hint: Flume streams Kafka to HDFS; MapReduce runs batch jobs [OK]

Common Mistakes:

• Using Kafka producer to write directly to HDFS (not supported)
• Confusing streaming with batch tools
• Manual upload instead of automated pipeline